Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prf eval #795

Closed
wants to merge 118 commits into from
Closed
Show file tree
Hide file tree
Changes from 41 commits
Commits
Show all changes
118 commits
Select commit Hold shift + click to select a range
cd46b44
added 25519 prime field
Oct 2, 2023
ce55114
Merge branch 'private-attribution:main' into largeprimefield
danielmasny Oct 2, 2023
e59b214
add files, make ec pub
Oct 2, 2023
1b552f4
last commit before working on shared curve points
Oct 2, 2023
9dc50c4
adding curve points + conversion
Oct 2, 2023
94597b1
from, into scalar for RP25519 + test
Oct 3, 2023
4da8014
test aritmetics for RP25519
Oct 3, 2023
fd6c28e
upgrade share<Fp25519> to share<curve point>
Oct 4, 2023
3c5683e
remove warning
Oct 4, 2023
b216481
add hash curve points
Oct 4, 2023
06f6eb1
implementation of PRF eval protocol
Oct 5, 2023
de5f32c
prf eval test passes
Oct 6, 2023
759f53a
Merge branch 'private-attribution:main' into PRF-Eval
danielmasny Oct 6, 2023
552cc62
fixed clippy complaints, panics when setting invalid curve points or …
Oct 9, 2023
fe3f130
remove comment
Oct 9, 2023
159cd2c
adding some of Martins suggestions
Oct 9, 2023
301a6e5
Alex suggestions, serialize, deserialize
Oct 16, 2023
9a1fa7c
Update src/ff/curve_points.rs
danielmasny Oct 16, 2023
8386d24
Alex suggestions, simplify u32, u64 macros
Oct 16, 2023
3468de5
Merge branch 'PRF-Eval' of https://github.com/danielmasny/ipa into PR…
Oct 16, 2023
5dc07a0
Update src/ff/curve_points.rs
danielmasny Oct 16, 2023
e7e6e63
Merge branch 'PRF-Eval' of https://github.com/danielmasny/ipa into PR…
Oct 16, 2023
e8d9d94
Merge branch 'private-attribution:main' into PRF-Eval
danielmasny Oct 16, 2023
94dda61
Merge branch 'PRF-Eval' of https://github.com/danielmasny/ipa into PR…
Oct 16, 2023
ea8da0d
simplify serde test
Oct 16, 2023
bcea6c6
simplify serde test
Oct 16, 2023
8bca660
error recursion limit
Oct 17, 2023
64dac77
Fix compiler recursion error
akoshelev Oct 17, 2023
884aeec
define WeakSharedValue, adding ipa-prf feature
Oct 17, 2023
a620e34
fix zero
Oct 17, 2023
55cef13
fmt
Oct 17, 2023
9736b71
fix clippy
Oct 17, 2023
2c8a19c
Merge branch 'private-attribution:main' into PRF-Eval
danielmasny Oct 17, 2023
b0ae43f
refactor prf_ipa in separate module
Oct 18, 2023
da13880
fmt
Oct 18, 2023
ecd1885
Add attribution window to OPRF aggregation
taikiy Oct 4, 2023
141a6b1
Optimize by removing redundant multiplications
taikiy Oct 19, 2023
946a9ee
save one more multiplication
taikiy Oct 20, 2023
a5d3587
inline the variables
taikiy Oct 20, 2023
89c86ec
Merge branch 'private-attribution:main' into PRF-Eval
danielmasny Oct 20, 2023
160ba83
adressing Alex's comments
Oct 20, 2023
8407938
fmt
Oct 20, 2023
43696a6
add comments
Oct 20, 2023
4268a05
fix lint
Oct 21, 2023
4208d1b
fmt
Oct 21, 2023
53d71f2
remove did_trigger_get_attributed and refactor attribution flag calcu…
taikiy Oct 23, 2023
1ce4281
1. Increasing max number of breakdowns we can support with OPRF
richajaindce Oct 24, 2023
034f556
Adding a oneshot for oprf
richajaindce Oct 24, 2023
2692af8
Now oprf can run for compact-gate as well
richajaindce Oct 24, 2023
2a798a3
Adding oprf handling in collect_steps.py
richajaindce Oct 24, 2023
8fa6745
refactoring collect_steps
richajaindce Oct 24, 2023
16c8943
Merge pull request #808 from taikiy/attribution_windowing
benjaminsavage Oct 25, 2023
d9d9672
Add test for no contribution + handle 512 breakdowns in test using Gf…
richajaindce Oct 24, 2023
5ea37ca
Supporting the opposite capping logic
benjaminsavage Oct 25, 2023
9fdad64
Merge pull request #815 from private-attribution/support_reverse_order
benjaminsavage Oct 25, 2023
0c5f464
Merge branch 'main' into minor_fixes
richajaindce Oct 25, 2023
c348906
Merge branch 'minor_fixes' into oneshot
richajaindce Oct 25, 2023
090d94f
Merge pull request #813 from richajaindce/minor_fixes
benjaminsavage Oct 25, 2023
dd07816
Peer feedback
richajaindce Oct 25, 2023
51e05fe
Some more fixes
richajaindce Oct 25, 2023
b396557
Merge pull request #814 from richajaindce/oneshot
benjaminsavage Oct 25, 2023
d8c0598
Some code refactor based on Taiki's feedback
richajaindce Oct 25, 2023
31a155c
OPRF test on real world infra
richajaindce Oct 26, 2023
9a59780
Merge pull request #818 from richajaindce/oneshot
benjaminsavage Oct 30, 2023
73468dc
Adding EventType in serialization and deserialization
richajaindce Oct 27, 2023
f77a8b4
Incorporate feedback
richajaindce Oct 30, 2023
2e4b237
Merge branch 'main' into real_world
richajaindce Oct 30, 2023
4d2469e
Make clippy happy
richajaindce Oct 30, 2023
0e9f41b
More fixes
richajaindce Oct 31, 2023
91905cd
Remove output.txt
richajaindce Oct 31, 2023
0747a6f
Rename function
richajaindce Oct 31, 2023
b4760d9
Merge pull request #822 from richajaindce/real_world
benjaminsavage Oct 31, 2023
2b50c45
Fix OPRF attribution and capping circuit
akoshelev Oct 31, 2023
b380eed
Merge pull request #825 from private-attribution/fix-oprf-attribution
benjaminsavage Nov 1, 2023
afeb72c
modulus conversion with retention
martinthomson Nov 1, 2023
6ea4069
Merge pull request #826 from martinthomson/retain-mod-convert
benjaminsavage Nov 1, 2023
da2ee44
Configure logging from env as well
akoshelev Oct 31, 2023
b4d10d6
Add send and receive trace spans
akoshelev Oct 31, 2023
9b7649b
added 25519 prime field
Oct 2, 2023
786d7c8
add files, make ec pub
Oct 2, 2023
c9cb5ee
last commit before working on shared curve points
Oct 2, 2023
31cbde8
adding curve points + conversion
Oct 2, 2023
6219b4c
from, into scalar for RP25519 + test
Oct 3, 2023
c65a93f
test aritmetics for RP25519
Oct 3, 2023
1acb869
upgrade share<Fp25519> to share<curve point>
Oct 4, 2023
644a6fc
remove warning
Oct 4, 2023
851f624
add hash curve points
Oct 4, 2023
45b2a46
implementation of PRF eval protocol
Oct 5, 2023
dd07d7a
prf eval test passes
Oct 6, 2023
e1cb4b7
fixed clippy complaints, panics when setting invalid curve points or …
Oct 9, 2023
b2f5c78
remove comment
Oct 9, 2023
8b0ab77
adding some of Martins suggestions
Oct 9, 2023
4c175bd
Alex suggestions, serialize, deserialize
Oct 16, 2023
537e7af
Alex suggestions, simplify u32, u64 macros
Oct 16, 2023
5325f6d
Update src/ff/curve_points.rs
danielmasny Oct 16, 2023
dd0aac0
Update src/ff/curve_points.rs
danielmasny Oct 16, 2023
5edee6f
simplify serde test
Oct 16, 2023
3fdd719
simplify serde test
Oct 16, 2023
5e65826
error recursion limit
Oct 17, 2023
b89d15a
Fix compiler recursion error
akoshelev Oct 17, 2023
b293fb5
define WeakSharedValue, adding ipa-prf feature
Oct 17, 2023
deeca7d
fix zero
Oct 17, 2023
729c87f
fmt
Oct 17, 2023
c9dc931
fix clippy
Oct 17, 2023
3aaeba4
refactor prf_ipa in separate module
Oct 18, 2023
04fe4d3
fmt
Oct 18, 2023
7db9376
adressing Alex's comments
Oct 20, 2023
7c211b7
fmt
Oct 20, 2023
4d5b105
add comments
Oct 20, 2023
39c7ced
fix lint
Oct 21, 2023
e5db397
fmt
Oct 21, 2023
c6564f0
Merge branch 'PRF-Eval' of https://github.com/danielmasny/ipa into PR…
Nov 2, 2023
0efd323
fix errors after merge
Nov 2, 2023
60d8ab6
fmt
Nov 2, 2023
a17eaa9
add flag
Nov 2, 2023
83d74cb
remove flag
Nov 2, 2023
f8d26c9
add flag
Nov 2, 2023
6c715ff
add flag descriptive gate
Nov 2, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ default = [
"tracing/max_level_trace",
"tracing/release_max_level_info",
"descriptive-gate",
"aggregate-circuit"
"aggregate-circuit",
"ipa-prf"
danielmasny marked this conversation as resolved.
Show resolved Hide resolved
]
cli = ["comfy-table", "clap"]
enable-serde = ["serde", "serde_json"]
Expand Down Expand Up @@ -41,6 +42,7 @@ compact-gate = ["ipa-macros/compact-gate"]
# Standalone aggregation protocol. We use IPA infra for communication
# but it has nothing to do with IPA.
aggregate-circuit = []
ipa-prf = ["descriptive-gate"]

[dependencies]
aes = "0.8.3"
Expand All @@ -54,6 +56,7 @@ clap = { version = "4.3.2", optional = true, features = ["derive"] }
comfy-table = { version = "7.0", optional = true }
config = "0.13.2"
criterion = { version = "0.5.1", optional = true, default-features = false, features = ["async_tokio", "plotters", "html_reports"] }
curve25519-dalek = "4.1.1"
danielmasny marked this conversation as resolved.
Show resolved Hide resolved
dashmap = "5.4"
dhat = "0.3.2"
embed-doc-image = "0.1.4"
Expand Down
2 changes: 2 additions & 0 deletions src/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,8 @@ pub enum Error {
InvalidReport(#[from] InvalidReportError),
#[error("unsupported: {0}")]
Unsupported(String),
#[error("Decompressing invalid elliptic curve point: {0}")]
DecompressingInvalidCurvePoint(String),
}

impl Default for Error {
Expand Down
239 changes: 239 additions & 0 deletions src/ff/curve_points.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,239 @@
use curve25519_dalek::{
ristretto::{CompressedRistretto, RistrettoPoint},
Scalar,
};
use generic_array::GenericArray;
use typenum::U32;

use crate::{
ff::{ec_prime_field::Fp25519, Serializable},
secret_sharing::{Block, WeakSharedValue},
};

impl Block for CompressedRistretto {
type Size = U32;
}

///ristretto point for curve 25519,
/// we store it in compressed format since it is 3 times smaller and we do a limited amount of
/// arithmetic operations on the curve points
///
/// We use ristretto points such that we have a prime order elliptic curve,
/// This is needed for the Dodis Yampolski PRF
///
/// decompressing invalid curve points will cause panics,
/// since we always generate curve points from scalars (elements in Fp25519) and
/// only deserialize previously serialized valid points, panics will not occur
/// However, we still added a debug assert to deserialize since values are sent by other servers
#[derive(Clone, Copy, PartialEq, Debug)]
pub struct RP25519(CompressedRistretto);

/// Implementing trait for secret sharing
impl WeakSharedValue for RP25519 {
type Storage = CompressedRistretto;
const BITS: u32 = 256;
const ZERO: Self = Self(CompressedRistretto([0_u8; 32]));
}

impl Serializable for RP25519 {
type Size = <<RP25519 as WeakSharedValue>::Storage as Block>::Size;

fn serialize(&self, buf: &mut GenericArray<u8, Self::Size>) {
*buf.as_mut() = self.0.to_bytes();
}

fn deserialize(buf: &GenericArray<u8, Self::Size>) -> Self {
debug_assert!(CompressedRistretto((*buf).into()).decompress().is_some());
RP25519(CompressedRistretto((*buf).into()))
akoshelev marked this conversation as resolved.
Show resolved Hide resolved
}
}

///## Panics
/// Panics when decompressing invalid curve point. This can happen when deserialize curve point
/// from bit array that does not have a valid representation on the curve
impl std::ops::Add for RP25519 {
type Output = Self;

fn add(self, rhs: Self) -> Self::Output {
Self((self.0.decompress().unwrap() + rhs.0.decompress().unwrap()).compress())
}
}

impl std::ops::AddAssign for RP25519 {
#[allow(clippy::assign_op_pattern)]
fn add_assign(&mut self, rhs: Self) {
*self = *self + rhs;
}
}

///## Panics
/// Panics when decompressing invalid curve point. This can happen when deserialize curve point
/// from bit array that does not have a valid representation on the curve
impl std::ops::Neg for RP25519 {
type Output = Self;

fn neg(self) -> Self::Output {
Self(self.0.decompress().unwrap().neg().compress())
}
}

///## Panics
/// Panics when decompressing invalid curve point. This can happen when deserialize curve point
/// from bit array that does not have a valid representation on the curve
impl std::ops::Sub for RP25519 {
type Output = Self;

fn sub(self, rhs: Self) -> Self::Output {
Self((self.0.decompress().unwrap() - rhs.0.decompress().unwrap()).compress())
}
}

impl std::ops::SubAssign for RP25519 {
#[allow(clippy::assign_op_pattern)]
fn sub_assign(&mut self, rhs: Self) {
*self = *self - rhs;
}
}

///Scalar Multiplication
/// allows to multiply curve points with scalars from Fp25519
///## Panics
/// Panics when decompressing invalid curve point. This can happen when deserialize curve point
/// from bit array that does not have a valid representation on the curve
impl std::ops::Mul<Fp25519> for RP25519 {
type Output = Self;

fn mul(self, rhs: Fp25519) -> RP25519 {
(self.0.decompress().unwrap() * Scalar::from(rhs))
.compress()
.into()
}
}

impl std::ops::MulAssign<Fp25519> for RP25519 {
#[allow(clippy::assign_op_pattern)]
fn mul_assign(&mut self, rhs: Fp25519) {
*self = *self * rhs;
}
}

impl From<Scalar> for RP25519 {
danielmasny marked this conversation as resolved.
Show resolved Hide resolved
fn from(s: Scalar) -> Self {
RP25519(RistrettoPoint::mul_base(&s).compress())
}
}

impl From<Fp25519> for RP25519 {
danielmasny marked this conversation as resolved.
Show resolved Hide resolved
fn from(s: Fp25519) -> Self {
RP25519(RistrettoPoint::mul_base(&s.into()).compress())
}
}

impl From<CompressedRistretto> for RP25519 {
fn from(s: CompressedRistretto) -> Self {
RP25519(s)
}
}

impl From<RP25519> for CompressedRistretto {
fn from(s: RP25519) -> Self {
s.0
}
}

///allows to convert curve points into unsigned integers, preserving high entropy
macro_rules! cp_hash_impl {
( $u_type:ty) => {
impl From<RP25519> for $u_type {
fn from(s: RP25519) -> Self {
use hkdf::Hkdf;
use sha2::Sha256;
let hk = Hkdf::<Sha256>::new(None, s.0.as_bytes());
danielmasny marked this conversation as resolved.
Show resolved Hide resolved
let mut okm = <$u_type>::MIN.to_le_bytes();
//error invalid length from expand only happens when okm is very large
hk.expand(&[], &mut okm).unwrap();
<$u_type>::from_le_bytes(okm)
}
}
};
}

cp_hash_impl!(u128);

cp_hash_impl!(u64);

/// implementing random curve point generation for testing purposes,
/// in the actual IPA protocol, we generate them from scalars, i.e. Fp25519
#[cfg(test)]
impl rand::distributions::Distribution<RP25519> for rand::distributions::Standard {
fn sample<R: crate::rand::Rng + ?Sized>(&self, rng: &mut R) -> RP25519 {
let mut scalar_bytes = [0u8; 64];
rng.fill_bytes(&mut scalar_bytes);
RP25519(RistrettoPoint::from_uniform_bytes(&scalar_bytes).compress())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would still prefer that we not compress unless transmitting data. The extra size is negligible and the uncompressed form is more efficient to use.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could implement the traits we need for a decompressed version of curve points. I would probably still need to define arithmetic operations on top of compressed points to satisfy the traits needed for secret shares & sending messages.

I think we should definitely store compressed points in memory to not waste any memory since it is the main bottleneck for running larger queries. I dont think operating on decompressed points would significantly reduce the amount of compress and decompress operations since currently I am only generate random exponents and then generate a curve point from these exponents. After that, there is one reveal operation which requires three additions over curvepoints (in the clear, not secret shared!) and then another exponentiation on top of the revealed curve point. So there are two unnecessary compress decompress operations in total per record (plus 2 decompress when you don't store your own shares in decompressed format, the compress you still need to do since you need to send your shares ).

}
}

#[cfg(all(test, unit_test))]
mod test {
use curve25519_dalek::{constants, scalar::Scalar};
use generic_array::GenericArray;
use rand::{thread_rng, Rng};
use typenum::U32;

use crate::{
ff::{curve_points::RP25519, ec_prime_field::Fp25519, Serializable},
secret_sharing::WeakSharedValue,
};

cp_hash_impl!(u32);

///testing serialize and deserialize
#[test]
fn serde_25519() {
let mut rng = thread_rng();
let input = rng.gen::<RP25519>();
let mut a: GenericArray<u8, U32> = [0u8; 32].into();
input.serialize(&mut a);
let output = RP25519::deserialize(&a);
assert_eq!(input, output);
}

///testing conversion from scalar to Fp25519 and curve point, i.e. RP25519
#[test]
fn scalar_to_point() {
let a = Scalar::ONE;
let b: RP25519 = a.into();
let d: Fp25519 = a.into();
let c: RP25519 = RP25519::from(d);
assert_eq!(b, RP25519(constants::RISTRETTO_BASEPOINT_COMPRESSED));
assert_eq!(c, RP25519(constants::RISTRETTO_BASEPOINT_COMPRESSED));
}

///testing simple curve arithmetics to check that `curve25519_dalek` library is used correctly
#[test]
fn curve_arithmetics() {
let mut rng = thread_rng();
let fp_a = rng.gen::<Fp25519>();
let fp_b = rng.gen::<Fp25519>();
let fp_c = fp_a + fp_b;
let fp_d = RP25519::from(fp_a) + RP25519::from(fp_b);
assert_eq!(fp_d, RP25519::from(fp_c));
assert_ne!(fp_d, RP25519(constants::RISTRETTO_BASEPOINT_COMPRESSED));
let fp_e = rng.gen::<Fp25519>();
let fp_f = rng.gen::<Fp25519>();
let fp_g = fp_e * fp_f;
let fp_h = RP25519::from(fp_e) * fp_f;
assert_eq!(fp_h, RP25519::from(fp_g));
assert_ne!(fp_h, RP25519(constants::RISTRETTO_BASEPOINT_COMPRESSED));
assert_eq!(RP25519::ZERO, fp_h * Scalar::ZERO.into());
}

///testing curve to unsigned integer conversion has entropy (!= 0)
#[test]
fn curve_point_to_hash() {
let mut rng = thread_rng();
let fp_a = rng.gen::<RP25519>();
assert_ne!(0u64, u64::from(fp_a));
assert_ne!(0u32, u32::from(fp_a));
}
}
Loading
Loading