Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Q: Again a stupid question... #618

Closed
neilyoung opened this issue Jan 31, 2018 · 7 comments
Closed

Q: Again a stupid question... #618

neilyoung opened this issue Jan 31, 2018 · 7 comments
Labels

Comments

@neilyoung
Copy link

... is there a way feed a webcam instance with RGB data? Like a "software cam"?

@sarxos
Copy link
Owner

sarxos commented Feb 1, 2018

Hi @neilyoung,

There is no such driver available in the release, but I have something similar available in tests. It's enough to copy the source code and use it in your project.

https://github.com/sarxos/webcam-capture/blob/master/webcam-capture/src/test/java/com/github/sarxos/webcam/ds/test/DummyDriver.java

https://github.com/sarxos/webcam-capture/blob/master/webcam-capture/src/test/java/com/github/sarxos/webcam/ds/test/DummyDevice.java

This DummyDevice renders red background with ball bouncing from the edges.

import javax.swing.JFrame;
import com.github.sarxos.webcam.Webcam;
import com.github.sarxos.webcam.WebcamPanel;
import com.github.sarxos.webcam.ds.test.DummyDriver;

public class DummyDriverExample {

	static {
		Webcam.setDriver(new DummyDriver());
	}

	public static void main(String[] args) {
		JFrame window = new JFrame("Dummy Test");
		window.add(new WebcamPanel(Webcam.getDefault()));
		window.setResizable(true);
		window.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
		window.pack();
		window.setVisible(true);
	}
}

image

If you want to feed your RGB data to the application and render as image from webcam then DummyDriver and DummyDevice are a good starting point. Just change getImage() method in DummyDevice to draw your RGB data on the returned image.

If you want to paint your RGB data on the real image from webcam, then WebcamImageTransformer interface can do that for you:

https://github.com/sarxos/webcam-capture/blob/master/webcam-capture/src/example/java/ImageTransformerExample.java

In this example transformer does nothing more than changing image to grayscale, but in the same way you can paint any RGB data on the image.

@sarxos sarxos added the question label Feb 1, 2018
@neilyoung
Copy link
Author

@sarxos Thanks, will give it a try

@neilyoung
Copy link
Author

neilyoung commented Feb 3, 2018

OK, have used this as a template. It works excellent. By using it I was able to write my own MJPEG decoder (tailored to the needs). I found, that the frame rate I can achieve (with 30 fps) is more or less exactly 30 fps on the sink, whereas your MjpegDriver is reaching 18 to 20 fps. Not sure, what it is, but if you like I can share my sources on a private channel.

Because my consuming process has a little problem being fed with one and the same frame several times (the using app is able to achieve 90 fps) I'm now refactoring the code so that "getImage" is hanging on a semaphore and just released on a new "real" frame.

I will conserve the current state so that I can send it to you if you like. The MIME decoder is pretty much nailed to the project, but it works great.

EDIT: I have now introduced thread sync means, so that getImage() is hung if no new frame is available. The frame rate now is comparable to your results... I suppose, you are also doing sort of sync (not checked). So you can forget about :)

But what I don't understand is this: I have several frame rate counters

  1. The source is supposed to deliver 30 fps
  2. The driver on reception side counts more or less exactly 30 fps after decoding
  3. The getImage() function in driver counts more or less exactly 30 fps, so there is an instance above that, which gets the 30 fps

So now: At application level I'm only having 15 fps... I know we had this discussion already once, but I can't remember exactly. Or I think I wasn't following your suggestion, because it seemed to complicated to me...

But as a matter of fact: 30fps after decoding, 30fps at driver getImage(). But only 15 fps after webcam.getImage. This needs to be resolved...

Here is the application code running in a separate thread. I have omitted all unnecessary stuff for clarity.

@Override
    public void run() {
      
        mWebcam.open(false);

        long t1 = System.currentTimeMillis();
        int frameCount = 0;
        while (mWebcam.isOpen()) {
            BufferedImage image = mWebcam.getImage();
            if (image == null) {
                break;
            }
            frameCount++;
            long t2 = System.currentTimeMillis();
            long delta = t2-t1+1;
            if (delta > 1000) {
                float fps = (float)frameCount/delta*1000;
                t1 = t2;
                frameCount = 0;
                System.out.println(String.format("APP FPS: %.2f", fps));  // Here not more than 15 fps
            }
        }
    }

@neilyoung
Copy link
Author

@sarxos This log will not tell you much, but this is what I see (and currently don't understand):

fps : Measured at driver level, whenever a new frame is decoded and made available
FPS: Measured at driver level, whenever a new image is retrieved via getImage()
APP FPS: Measured at app level as shown above.

fps: 29.92
FPS: 29.92
APP FPS: 14.90
fps: 29.97
FPS: 29.97
APP FPS: 14.82
fps: 29.64
FPS: 29.62
APP FPS: 14.87
fps: 30.97
FPS: 30.10
APP FPS: 14.86
fps: 30.42
FPS: 30.30
APP FPS: 15.31
fps: 29.01
FPS: 29.93
APP FPS: 15.00
fps: 30.75
FPS: 30.78
APP FPS: 14.86
fps: 29.25
FPS: 29.27
APP FPS: 14.84
fps: 21.09
FPS: 20.21
APP FPS: 15.46
fps: 39.39
FPS: 39.42
APP FPS: 14.48
fps: 31.31
FPS: 31.31
APP FPS: 15.01
fps: 29.62
FPS: 29.62
APP FPS: 15.72
fps: 29.79
FPS: 29.79
APP FPS: 14.46
fps: 29.24
FPS: 29.24


@neilyoung
Copy link
Author

neilyoung commented Feb 3, 2018

OK, giving up for today. Don't get it really. Here is a raw trace of one particular second, which I don't understand:

Explanation: The returnFrame notify is issued, when a new frame has been decoded and is available as image. The same time the driver triggers a sync object

synchronized (syncLock) {
            deliveredFramecount++;
            Util.logger.trace("returnFrame notify");
            syncLock.notify();
}

The getImage leaveWait trace is issued in the driver's getImage() function. The function waits for the release of the object like so:

            Util.logger.trace("getImage enter wait");
            synchronized (syncLock) {
                syncLock.wait();
            }
            Util.logger.trace("getImage leave wait");

So as far as I can see each returnFrame notify is immediately followed by a getImage leave wait which returns the image and immediately calls getImage enter wait wait again, which means to me, that the getImage() is called in a loop. OK for me.

However, only every 2nd delivery ends up in a Got frame trace, which is issued by an application thread, which calls webcam.getImage() in a loop. In fact the trace totally confirms the weird observation, that I can't achieve full 30 fps at app level, just the half.

2018-02-03 23:46:11,812 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:11,812 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:11,812 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:11,824 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:11,824 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:11,825 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:11,825 Capture 63 TRACE [Thread-30]: Got frame
2018-02-03 23:46:11,873 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:11,873 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:11,873 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:11,886 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:11,887 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:11,887 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:11,887 Capture 63 TRACE [Thread-30]: Got frame
2018-02-03 23:46:11,939 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:11,939 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:11,939 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:11,953 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:11,953 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:11,953 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:11,953 Capture 63 TRACE [Thread-30]: Got frame
2018-02-03 23:46:12,005 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:12,005 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:12,006 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:12,017 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:12,017 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:12,017 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:12,017 Capture 63 TRACE [Thread-30]: Got frame
2018-02-03 23:46:12,072 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:12,072 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:12,072 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:12,086 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:12,086 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:12,086 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:12,086 Capture 63 TRACE [Thread-30]: Got frame
2018-02-03 23:46:12,147 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:12,147 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:12,147 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:12,157 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:12,158 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:12,158 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:12,158 Capture 63 TRACE [Thread-30]: Got frame
2018-02-03 23:46:12,198 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:12,198 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:12,198 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:12,204 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:12,204 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:12,204 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:12,204 Capture 63 TRACE [Thread-30]: Got frame
2018-02-03 23:46:12,264 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:12,264 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:12,264 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:12,270 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:12,270 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:12,270 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:12,270 Capture 63 TRACE [Thread-30]: Got frame
2018-02-03 23:46:12,329 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:12,329 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:12,329 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:12,336 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:12,337 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:12,337 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:12,337 Capture 63 TRACE [Thread-30]: Got frame
2018-02-03 23:46:12,400 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:12,400 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:12,401 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:12,410 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:12,410 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:12,410 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:12,411 Capture 63 TRACE [Thread-30]: Got frame
2018-02-03 23:46:12,417 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:12,417 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:12,419 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:12,474 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:12,475 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:12,475 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:12,529 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:12,530 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:12,530 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:12,531 Capture 63 TRACE [Thread-30]: Got frame
2018-02-03 23:46:12,539 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:12,539 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:12,541 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:12,595 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:12,596 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:12,596 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:12,602 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:12,602 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:12,602 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:12,602 Capture 63 TRACE [Thread-30]: Got frame
fps: 29.97
2018-02-03 23:46:12,665 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:12,666 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
FPS: 29.97
2018-02-03 23:46:12,666 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:12,672 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:12,672 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:12,672 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:12,672 Capture 63 TRACE [Thread-30]: Got frame
2018-02-03 23:46:12,727 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:12,727 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:12,727 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:12,736 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:12,736 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:12,736 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
2018-02-03 23:46:12,737 Capture 63 TRACE [Thread-30]: Got frame
2018-02-03 23:46:12,760 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify
2018-02-03 23:46:12,760 MJPEGDirectDevice 202 TRACE [atomic-processor-1]: getImage leave wait
2018-02-03 23:46:12,762 MJPEGDirectDevice 196 TRACE [atomic-processor-1]: getImage enter wait
APP FPS: 13.90
2018-02-03 23:46:12,826 MJPEGDirectDevice 68 TRACE [Thread-27]: returnFrame notify

@neilyoung
Copy link
Author

neilyoung commented Feb 4, 2018

Found our old conversation: #599. Still not convinced that it has to be that complicated. Thinking about a direct frame callback from driver to application...

And webcam.open(true/false) makes no difference

@neilyoung
Copy link
Author

@sarxos Finally found a better way: Dropped WebcamPanel, doing my own drawing in the capture thread. Can be closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants