Skip to content

Commit

Permalink
Update index.html
Browse files Browse the repository at this point in the history
  • Loading branch information
@aashishrai3799
aashishrai3799 authored Jul 30, 2024
1 parent c9ad387 commit 22874be
Showing 1 changed file with 142 additions and 61 deletions.
203 changes: 142 additions & 61 deletions index.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -93,68 +93,10 @@ <h1 class="title is-1 publication-title">EgoSonics: Generating Synchronized Audi
</div>
</section>

<section class="hero teaser">
<div class="container is-max-desktop">
<div class="hero-body">
<!-- <img src="./static/images/pull_figure.png" alt="teaser" style="width:80%"> -->
<!-- centered iamge -->
<img src="./assets/teaser.jpg" alt="teaser" style="display: block; margin-left: auto; margin-right: auto; width: 100%;">
<br>
<video poster="" id="teaser" autoplay controls muted loop playsinline height="100%">
<source src="./assets/videos/teaser.mp4" type="video/mp4">
</video>
<br>
<h2 class="subtitle has-text-centered">
We present EgoSonics, a method to synthesize audio tracks conditioned on silent in-the-wild egocentric videos.
Our method operate on videos at 30 fps to synthesize audio that is semantically meaningful and synchronized
with events in the video (“dish hits the sink” or “lawn mower turned off”).
</h2>
</div>
</div>
</section>


<!-- <section class="hero is-light is-small">
<div class="hero-body">
<div class="container">
<div id="results-carousel" class="carousel results-carousel">
<div class="item item-steve">
<video poster="" id="steve" autoplay controls muted loop playsinline height="100%">
<source src="./static/videos/steve.mp4"
type="video/mp4">
</video>
</div>
</div>
</div>
</div>
</section> -->


<section class="section">
<div class="container is-max-desktop">
<!-- Abstract. -->
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3">Abstract</h2>
<div class="content has-text-justified">
<p>
We introduce EgoSonics, a method to generate semantically meaningful and synchronized audio tracks conditioned on silent egocentric videos.
Generating audio for silent egocentric videos could open new applications in virtual reality, assistive technologies, or for augmenting existing datasets.
Existing work has been limited to domains like speech, music, or impact sounds and cannot easily capture the broad range of audio frequencies found in
egocentric videos. EgoSonics addresses these limitations by building on the strength of latent diffusion models for condi- tioned audio synthesis.
We first encode and process audio and video data into a form that is suitable for generation. The encoded data is used to train our model to generate
audio tracks that capture the semantics of the input video. Our proposed SyncroNet builds on top of ControlNet to provide control signals that enables
temporal synchronization to the synthesized audio. Extensive evaluations show that our model outperforms existing work in audio quality, and in our
newly proposed synchronization evaluation method. Furthermore, we demonstrate downstream applications of our model in improving video summarization.

</p>
</div>
</div>
</div>
<!--/ Abstract. -->
</div>
</section>




Expand All @@ -179,7 +121,7 @@ <h2 class="title is-3">Generated Samples from EgoSonics (our method)</h2>

<div class="column is-one-third">
<video height="100%" controls="">
<source src="./assets/videos/example3.mp4" type="video/mp4">
<source src="./assets/videos/example10.mp4" type="video/mp4">
</video>
</div>

Expand Down Expand Up @@ -215,7 +157,7 @@ <h2 class="title is-3">Generated Samples from EgoSonics (our method)</h2>

<div class="column is-one-third">
<video height="100%" controls="">
<source src="./assets/videos/example8.mp4" type="video/mp4">
<source src="./assets/videos/example3.mp4" type="video/mp4">
</video>
</div>

Expand All @@ -237,6 +179,145 @@ <h2 class="title is-3">Generated Samples from EgoSonics (our method)</h2>






<!-- <section class="hero is-light is-small">
<div class="hero-body">
<div class="container">
<div id="results-carousel" class="carousel results-carousel">
<div class="item item-steve">
<video poster="" id="steve" autoplay controls loop height="100%">
<source src="./assets/videos/example1.mp4"
type="video/mp4">
</video>
</div>
<div class="item item-chair-tp">
<video poster="" id="chair-tp" autoplay controls loop height="100%">
<source src="./assets/videos/example2.mp4"
type="video/mp4">
</video>
</div>
<div class="item item-fullbody">
<video poster="" id="fullbody" autoplay controls loop height="100%">
<source src="./assets/videos/example6.mp4"
type="video/mp4">
</video>
</div>
<div class="item item-fullbody">
<video poster="" id="fullbody" autoplay controls loop height="100%">
<source src="./assets/videos/example4.mp4"
type="video/mp4">
</video>
</div>
<div class="item item-fullbody">
<video poster="" id="fullbody" autoplay controls loop height="100%">
<source src="./assets/videos/example5.mp4"
type="video/mp4">
</video>
</div>
<div class="item item-fullbody">
<video poster="" id="fullbody" autoplay controls loop height="100%">
<source src="./assets/videos/example3.mp4"
type="video/mp4">
</video>
</div>
<div class="item item-fullbody">
<video poster="" id="fullbody" autoplay controls loop height="100%">
<source src="./assets/videos/example10.mp4"
type="video/mp4">
</video>
</div>
<div class="item item-fullbody">
<video poster="" id="fullbody" autoplay controls loop height="100%">
<source src="./assets/videos/example9.mp4"
type="video/mp4">
</video>
</div>
<div class="item item-shiba">
<video poster="" id="shiba" autoplay controls loop height="100%">
<source src="./assets/videos/example7.mp4"
type="video/mp4">
</video>
</div>
</div>
</div>
</div>
</section> -->












<section class=" is-centered has-text-centered">
<div class="container is-max-desktop">
<div class="hero-body">
<h2 class="title is-3">Overview</h2>
<!-- <img src="./static/images/pull_figure.png" alt="teaser" style="width:80%"> -->
<!-- centered iamge -->
<img src="./assets/teaser.jpg" alt="teaser" style="display: block; margin-left: auto; margin-right: auto; width: 100%;">
<br>
<video poster="" id="teaser" autoplay controls muted loop playsinline height="100%">
<source src="./assets/videos/teaser.mp4" type="video/mp4">
</video>
<br>
<h2 class="subtitle has-text-centered">
We present EgoSonics, a method to synthesize audio tracks conditioned on silent in-the-wild egocentric videos.
Our method operate on videos at 30 fps to synthesize audio that is semantically meaningful and synchronized
with events in the video (“dish hits the sink” or “lawn mower turned off”).
</h2>
</div>
</div>
</section>



<section class="section">
<div class="container is-max-desktop">
<!-- Abstract. -->
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3">Abstract</h2>
<div class="content has-text-justified">
<p>
We introduce EgoSonics, a method to generate semantically meaningful and synchronized audio tracks conditioned on silent egocentric videos.
Generating audio for silent egocentric videos could open new applications in virtual reality, assistive technologies, or for augmenting existing datasets.
Existing work has been limited to domains like speech, music, or impact sounds and cannot easily capture the broad range of audio frequencies found in
egocentric videos. EgoSonics addresses these limitations by building on the strength of latent diffusion models for condi- tioned audio synthesis.
We first encode and process audio and video data into a form that is suitable for generation. The encoded data is used to train our model to generate
audio tracks that capture the semantics of the input video. Our proposed SyncroNet builds on top of ControlNet to provide control signals that enables
temporal synchronization to the synthesized audio. Extensive evaluations show that our model outperforms existing work in audio quality, and in our
newly proposed synchronization evaluation method. Furthermore, we demonstrate downstream applications of our model in improving video summarization.

</p>
</div>
</div>
</div>
<!--/ Abstract. -->
</div>
</section>








<section class="section">
<!-- qualitative examples on VR games two column format -->
<div class="columns is-centered has-text-centered">
Expand Down Expand Up @@ -435,4 +516,4 @@ <h2 class="title">BibTeX</h2>



</body></html>
</body></html>

0 comments on commit 22874be

Please sign in to comment.