How is the XML encoded in the RTP payload for ONVIF metadata streaming?

[xtrdustin]

We're initiating an RTSP connection, sending a PLAY request, and we start getting RTP packets, as described in the ONVIF Streaming Spec. I believe I'm parsing the headers correctly, as they have a payload type of 96, the first available payload type; other header fields look right. However, the payload of these RTP packets just appears to be binary data. It doesn't look like XML or text at all. Is there some other step to get the XML out of that RTP data, like decompression or something like that?

[xtrdustin]

Tagging @svefredrik, as you've answered questions about ONVIF metadata streaming in the past. Any info you can give me about the encoding issue would be helpful.

[svefredrik]

Can you include a Wireshark trace here? I can't say what's happening exactly without a trace but I believe that 96 is normally used for video so I think that the profile you are using just sets up a video stream, no metadata stream.

[xtrdustin]

Hi Fredrik, thanks for getting back to me.

Wireshark doesn't currently work on my VPN, but I had my program log every byte it received over the socket connection from the server to a file. It would make sense that it's video data given the size (it's over a MB for maybe 30 seconds of data). I'll upload it via your help desk, as I don't want to post it publicly if it's video of our test area. (If needed, I can capture the client side when I have the camera back in the next day or two, but I can tell you more about the requests I'm sending if you have questions.)

I don't know why it would be sending video though. I created a new profile (profile token is "metadata"), associated it with a MetadataConfiguration, and set position and move status to true. This was done through ONVIF, but I also looked at the profile's settings in the camera's web interface. The "metadata" profile didn't appear to have video streaming turned on.

The request URL that I'm using is rtsp://[camera ip]/onvif-media/media.amp?profile=metadata&sessiontimeout=60&streamtype=unicast

Edited to add: I created a ticket for uploading the file (Case 02026701), but I don't think the file is uploading successfully.

[svefredrik]

I don't need a full trace. I'm mainly interested in the SDP answer to RTSP DESCRIBE. If there's a metadata track there or not. It would also be good to see the listing of the profile that you use, i.e. response to GetProfiles.

[svefredrik]

And leave VideoEncoder=null

[xtrdustin]

I associated the profile with a VideoSourceConfiguration and a VideoAnalyticsConfiguration. It doesn't seem to have helped though. I'm still just seeing binary data in the stream.

Here's the updated profile:

=== PROFILE for token "metadata" ===
org.onvif.ver10.schema.Profile@87220f1[
   name=metadata apg,
   videoSourceConfiguration=org.onvif.ver10.schema.VideoSourceConfiguration@774013dc[
      sourceToken=0,
      bounds=org.onvif.ver10.schema.IntRectangle@229cb4d8[
         x=0,
         y=0,
         width=1920,
         height=1080
      ],
      any=<null>,
      extension=org.onvif.ver10.schema.VideoSourceConfigurationExtension@76e9eed8[
         rotate=org.onvif.ver10.schema.Rotate@30ef32eb[
            mode=ON,
            degree=0,
            extension=<null>,
            otherAttributes={}
         ],
         extension=<null>
      ],
      otherAttributes={},
      name=user0,
      useCount=3,
      token=0
   ],
   audioSourceConfiguration=<null>,
   videoEncoderConfiguration=<null>,
   audioEncoderConfiguration=<null>,
   videoAnalyticsConfiguration=org.onvif.ver10.schema.VideoAnalyticsConfiguration@bb5f9d[
      analyticsEngineConfiguration=org.onvif.ver10.schema.AnalyticsEngineConfiguration@391dfe7c[
         analyticsModule=<null>,
         extension=<null>,
         otherAttributes={}
      ],
      ruleEngineConfiguration=org.onvif.ver10.schema.RuleEngineConfiguration@702cfbde[
         rule=<null>,
         extension=<null>,
         otherAttributes={}
      ],
      any=<null>,
      otherAttributes={},
      name=analytics0,
      useCount=1,
      token=0
   ],
   ptzConfiguration=<null>,
   metadataConfiguration=org.onvif.ver10.schema.MetadataConfiguration@7d3bcd3[
      ptzStatus=org.onvif.ver10.schema.PTZFilter@5dce5c03[
         status=true,
         position=true,
         otherAttributes={}
      ],
      events=<null>,
      analytics=false,
      multicast=org.onvif.ver10.schema.MulticastConfiguration@b76b7d8[
         address=org.onvif.ver10.schema.IPAddress@1c135f63[
            type=I_PV_4,
            iPv4Address=0.0.0.0,
            iPv6Address=<null>
         ],
         port=0,
         ttl=5,
         autoStart=false,
         any=<null>,
         otherAttributes={}
      ],
      sessionTimeout=PT60S,
      any=<null>,
      analyticsEngineConfiguration=<null>,
      extension=<null>,
      compressionType=<null>,
      otherAttributes={},
      name=user0,
      useCount=1,
      token=0
   ],
   extension=<null>,
   token=metadata,
   fixed=false,
   otherAttributes={}
]

===============================

And here's the current DESCRIBE request/response:

===============
Sending:
DESCRIBE rtsp://10.1.10.42/onvif-media/media.amp?profile=metadata&sessiontimeout=60&streamtype=unicast RTSP/1.0
Authorization: Digest username="onviftest", realm="AXIS_WS_B8A44F212011", nonce="0002c46dY569492525f7fbbbf9a5b7d03cb5145fb705eca", uri="rtsp://10.1.10.42/onvif-media/media.amp?profile=metadata&sessiontimeout=60&streamtype=unicast", response="[omitted]"
CSeq: 1


===================
Received bytes:
RTSP/1.0 200 OK
CSeq: 1
Content-Type: application/sdp
Content-Base: rtsp://10.1.10.42/onvif-media/media.amp/
Server: GStreamer RTSP server
Date: Thu, 03 Nov 2022 17:04:45 GMT
Content-Length: 751

v=0
o=- 10956685192104015360 1 IN IP4 10.1.10.42
s=Session streamed with GStreamer
i=rtsp-server
t=0 0
a=tool:GStreamer
a=type:broadcast
a=range:npt=now-
a=control:rtsp://10.1.10.42/onvif-media/media.amp?profile=metadata&sessiontimeout=60&streamtype=unicast
m=video 0 RTP/AVP 96
c=IN IP4 0.0.0.0
b=AS:50000
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1;profile-level-id=64002a;sprop-parameter-sets=Z2QAKq0AxSAeAIn5cBbgICA+AAbd0AM3+YAeJEVA,aO48sA==
a=ts-refclk:local
a=mediaclk:sender
a=recvonly
a=control:rtsp://10.1.10.42/onvif-media/media.amp/stream=0?profile=metadata&sessiontimeout=60&streamtype=unicast
a=framerate:60.000000
a=transform:1.000000,0.000000,0.000000;0.000000,1.000000,0.000000;0.000000,0.000000,1.000000

===============

[svefredrik]

There seems to be a problem with using a profile that only has Metadata. I don't have time right now to investigate it further and next week I'm away on a business trip. You can try a workaround if you want, if you add "&video=0&ptz=all" to the request URL so that it becomes: "rtsp://10.1.10.42/onvif-media/media.amp?profile=metadata&sessiontimeout=60&streamtype=unicast&video=0&ptz=all"
This should disable video and enable PTZ.

You can try with GStreamer first if you have that:
gst-launch-1.0 rtspsrc location="rtsp://user:[email protected]/onvif-media/media.amp?profile=metadata&sessiontimeout=60&streamtype=unicast&video=0&ptz=all" ! fdsink

[xtrdustin]

Thanks, I'm getting a feed of XML now. Adding those parameters to the URL should work for us until a permanent fix is available.

[svefredrik]

Great, I'll have a look when I'm back in office!

[svefredrik]

Please also include the GetProfiles response

[xtrdustin]

Will do.

[xtrdustin]

Here's the GetProfile response for "metadata":

=== PROFILE for token "metadata" ===
org.onvif.ver10.schema.Profile@405603b6[
name=metadata apg,
videoSourceConfiguration=<null>,
audioSourceConfiguration=<null>,
videoEncoderConfiguration=<null>,
audioEncoderConfiguration=<null>,
videoAnalyticsConfiguration=<null>,
ptzConfiguration=<null>,
metadataConfiguration=org.onvif.ver10.schema.MetadataConfiguration@48d31d25[
ptzStatus=org.onvif.ver10.schema.PTZFilter@46188a89[
status=true,
position=true,
otherAttributes={}
],
events=<null>,
analytics=false,
multicast=org.onvif.ver10.schema.MulticastConfiguration@6f65a203[
address=org.onvif.ver10.schema.IPAddress@1d0b447b[
type=I_PV_4,iPv4Address=0.0.0.0,iPv6Address=<null>
],
port=0,
ttl=5,
autoStart=false,
any=<null>,
otherAttributes={}
],
sessionTimeout=PT60S,
any=<null>,
analyticsEngineConfiguration=<null>,
extension=<null>,
compressionType=<null>,
otherAttributes={},
name=user0,
useCount=1,
token=0
],
extension=<null>,
token=metadata,
fixed=false,
otherAttributes={}
]


===============================

Edit: Added newlines for readability.

[svefredrik]

You have to set a videoSourceConfiguration for the profile otherwise the MetadataConfiguration is pointless

[svefredrik]

You also need to set videoAnalyticsConfiguration

[karhe534]

Since PTZ metadata is wanted, a PTZ configuration also needs to be associated with the media profile.
Solely a MetadataConfiguration is not enough.
I think that is the problem. The ONVIF media profile in use does not seem to have any PTZ configuration associated with it.
(ptzConfiguration=)
Try associating a PTZConfiguration along with the MetadataConfiguration in the profile and see if that doesn't solve the problem!

See Media Service Specification section '5.2.8 AddPTZConfiguration'
"Adding a PTZConfiguration to a media profile means that streams using that media profile can contain PTZ
status (in the metadata), and that the media profile can be used for controlling PTZ movement [...]"

AnalyticsConfiguration and VideoSourceConfiguration are needed if you want to stream analytics metadata, so should not be needed for only PTZ metadata.

[xtrdustin]

This seems to work as well, and appending the "&video=0&ptz=all" string to the URL was no longer necessary (output looks the same as when that string was appended). Thanks!