My depth model was obsessed with the sky. Here's how I made it care about cars.

So, quick story while you finish your lunch.

For my M.Tech thesis I've been working on a model that looks at one ordinary camera image and does two jobs at once:

1. Finds the objects: cars, pedestrians, cyclists (object detection).
2. Guesses how far away everything is: a distance value for every single pixel (monocular depth estimation).

Think of a dashcam that can say "there's a car, and it's about 14 metres ahead." That's the dream.

The paper just got accepted at ICTCS 2026 (Springer LNNS), so I figured I'd tell you what we actually found without all the equations.

The problem: the model was studying the wrong chapter

Here's the thing nobody tells you about depth models.

When you train one, the loss function checks the predicted distance at every pixel and treats them all equally. Sounds fair, right?

Now look at a typical driving photo. Most of it is road, sky, buildings and trees. The cars, the stuff you'd actually crash into, take up only a small slice of the image.

So the model gets really good at estimating how far away the sky is... and is kind of "meh" about the car in front of you.

It's like preparing for an exam by memorising the index because it has the most pages.

The fix: tell the loss where the important stuff is

We already have bounding boxes for every object, because the detection half of the model needs them anyway.

So the idea is almost embarrassingly simple:

Pixels inside an object's box count 5× more in the depth loss. Everything else counts as normal.

No new labels. No extra network. One weighted loss. We call it object-aware depth supervision.

But there's a catch. Our model is juggling two tasks, detection and depth, and if you crank up depth you might quietly break detection. Tuning that balance by hand is painful.

So we added uncertainty weighting (an idea from Kendall et al., 2018). Instead of us guessing how much each task should matter, the network learns the balance during training.

The setup underneath is boring on purpose: a ResNet-18 encoder shared by both tasks, a U-Net style decoder for depth and a CenterNet head for detection. One network, two outputs.

Did it work?

Yep. On KITTI (real driving footage), here's the depth error on objects:

• Object-aware loss alone: 8.7% lower error on cars.
• Object-aware + uncertainty weighting: 19.6% lower.

And the fun part: the combination wasn't just "a bit better". Each trick helps on its own, but together they help a lot more. The whole-image depth error dropped too (0.134 → 0.115), so we didn't make the background worse to make the cars better.

We also ran it on Virtual KITTI 2 (a synthetic copy of KITTI) with five different random seeds, because one lucky run proves nothing:

Detection accuracy (mAP) on that dataset stayed basically flat: 0.581 → 0.580.

I also tried it on other backbones (MobileNetV3 and EfficientNet-B0). There were fewer runs there, but the same pattern showed up.

The honest part: where it didn't help much

I want to be upfront about this, because it's actually the most interesting result.

We broke the improvement down by object type:

Near and distant objects improved most of the time. Occluded objects (a car half-hidden behind another car) barely did better than a coin flip.

Why? Think about it: the box around a hidden car also contains pixels of the car in front of it. So we're telling the model "pay 5× attention here!" while pointing at the wrong car. Oops.

Also, on real KITTI there's a small drop in detection mAP (0.161 → 0.151). It's not statistically significant, but I can't promise it's zero either. So: big depth win, maybe a tiny detection cost.

That occlusion problem is exactly what I'm working on next.

"Will it run on a car, though?"

One reviewer asked this, which is fair. We don't have a Jetson (yet), so I did the next best thing: tested on a Mac mini as an edge-device stand-in.

Frames per second on a Mac mini: one joint network is 1.55 to 1.74 times faster than two separate networks on CPU and GPU, and half the size (58 MB vs 115 MB)

One joint network vs running two separate models (one for detection, one for depth):

• 1.55–1.74× faster.
• Half the size: 58 MB vs 115 MB.

Makes sense: both tasks share the same encoder, so you pay for "looking at the image" only once.

Behind the scenes (a.k.a. the part that almost killed me)

Okay, real talk. The ML was the easy part.

A few days before the camera-ready deadline, the workstation I was training on started randomly crashing. Segfaults. In waves. After roughly two hours of uptime, every single time.

I checked everything:

• GPU drivers? Fine.
• Disks? SMART says healthy.
• RAM? memtester passed.
• Corrupt data? Nope.

Best guess: a degrading CPU. That model family has had well-known stability issues. Reboot it and you get about two clean hours before the chaos starts again.

So I did what any sleep-deprived grad student would do: moved the final runs to a Mac mini, ported the training to Apple's MPS backend, and let it grind overnight at about 8 minutes per epoch. It finished the missing KITTI runs, and the numbers matched the server's.

Lesson learned: always have a plan B machine, and save checkpoints like your thesis depends on it, because it does.

TL;DR

• Depth models treat every pixel equally, so they under-invest in the objects that matter.
• Weighting object pixels 5× fixes a lot of that, for free.
• Adding learned uncertainty weighting makes it even better: −19.6% object depth error on KITTI.
• It still struggles with occluded objects, which is my next problem to solve.
• One joint network is ~1.6× faster and half the size of two separate ones.

Paper: "Object-Aware Depth Supervision for Joint Monocular Depth Estimation and Object Detection", accepted at ICTCS 2026 (Springer LNNS). Co-authors: Venkateswara Rao Torlapati, Y. Venkata Raghava Rao, Veeraswami Pilli and Prasad Tadi, Sree Vahini Institute of Science and Technology. I'll link the DOI once the proceedings are out.

If you work on depth, detection or anything self-driving-adjacent, I'd love to hear how you deal with occlusion. Drop a comment!