The issue seems to be due to a bad end of line and the way the boundary is retrieved. According to a RFC2046 quote taken from a SO answer:

The Content-Type field for multipart entities requires one parameter, “boundary”. The boundary delimiter line is then defined as a line consisting entirely of two hyphen characters (“-“, decimal value 45) followed by the boundary parameter value from the Content-Type header field, optional linear whitespace, and a terminating CRLF.

The problem lies precisely on two points: the end of line type and the two hyphens preceding the boundary parameter value.

End of lines

Since your code doesn’t show accurately the value of byteString, I tried both LF (n) and CRLF (rn) end of lines to see what will happen.

It appears the issue is reproduced when a bad end of line – i.e. not CRLF – is right before the last boundary, as shown below:

String byteString=
    "-----------------------------149742642616556rn" +
    "Content-Disposition: form-data; name="file"; filename="test.txt"rn" +
    "Content-Type: text/plain; charset=UTF-8rn" +
    "rn" +
    "testrn" + // <-- only n here lead to a MalformedStreamException
    "-----------------------------149742642616556--rn";

It sounds like the MultipartStream fails to parse the begin of the boundary, since it doesn’t catch a right end of line (CRLF) on the previous line. So, I you used LF terminators, you should replace them by CRLF ones.

Boundary format

The RFC tells that a boundary delimiter is two hyphens + boundary parameter + CRLF. Your regexp doesn’t catch only the boundary parameter value, it also includes the two hyphens. So I replaced this part:

// capturing group = boundary parameter value
String regexp="(?m)\A--(-*\d+)$";
// [...]
while (matcher.find()) {
    boundary = matcher.group(1);
    // [...]
}

Working code

Runnable as a MCVE

The code you’ll find below can be run in a console without Tomcat. Only commons-fileupload-1.3.3-bin.tar.gz and commons-io-2.6-bin.tar.gz are needed.

To view what’s parsed by the MultipartStream, I temporarily replaced bos by System.out in the readBodyData() call (as told in the comments).

• To compile:

  javac Test.java -classpath ./commons-fileupload-1.3.3-bin/commons-fileupload-1.3.3.jar 
  

• To run:

  java -classpath ./commons-fileupload-1.3.3-bin/commons-fileupload-1.3.3.jar:./commons-io-2.6/commons-io-2.6.jar:. Test
  

The code itself

import java.util.regex.*;
import java.io.*;
import org.apache.commons.fileupload.*;

public class Test {
    public final static void main(String[] argv) {
    String byteString=
        "-----------------------------149742642616556rn" +
        "Content-Disposition: form-data; name="file"; filename="test.txt"rn" +
        "Content-Type: text/plain; charset=UTF-8rn" +
        "rn" +
        "testrn" + // <-- only n here lead to a MalformedStreamException
        "-----------------------------149742642616556--rn";

    String regexp="(?m)\A--(-*\d+)$"; // edited regexp to catch the right boundary

    Pattern pattern = Pattern.compile(regexp);
    Matcher matcher = pattern.matcher(byteString);
    String boundary = null;
    String contentType=null;
    while (matcher.find()) {
        boundary = matcher.group(1);
        contentType = "multipart/form-data; boundary="" + boundary + """;
    }

    System.out.println("boundary = "" + boundary + """);

    @SuppressWarnings("deprecation")
        org.apache.commons.fileupload.MultipartStream multipartStream =
        new org.apache.commons.fileupload.MultipartStream
        (new ByteArrayInputStream(byteString.getBytes()), boundary.getBytes());
     ByteArrayOutputStream bos = new ByteArrayOutputStream();

    try {
        // Use the commented line instead the following one
        // To see what the multipartStream is reading (for debug)
        // multipartStream.readBodyData(System.out);
        multipartStream.readBodyData(bos);
    } catch (MultipartStream.MalformedStreamException e) {
        System.out.println("Malformed Exception " + e.getMessage());
    } catch (IOException e) {
        System.out.println(e.getMessage());
    }
    byte[] byteBody = bos.toByteArray();

    // Displaying the body read
    for(byte c : byteBody) {
        System.out.format("%c", c);
    }
    System.out.println();
    }
}

Output:

boundary = "---------------------------149742642616556"
-----------------------------149742642616556
Content-Disposition: form-data; name="file"; filename="test.txt"
Content-Type: text/plain; charset=UTF-8

test