Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More friendly host apis #224

Merged
merged 63 commits into from
Jan 5, 2024
Merged
Changes from 1 commit
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
f89a595
add wasm_witness
xgaozoyoe Nov 12, 2023
6b40e08
support standard env in cli
xgaozoyoe Nov 17, 2023
c5a8196
change wasm_witness_inject to wasm_witness_insert
xgaozoyoe Nov 17, 2023
a0d040c
support wasm_dbg_char
xgaozoyoe Nov 19, 2023
1d1dbce
move witness api into external foreign host
xgaozoyoe Nov 20, 2023
b8b37b5
fix: add missing files
xgaozoyoe Nov 20, 2023
89981fc
use better merkle circuit from zkwasm-host-circuits
xgaozoyoe Nov 20, 2023
f0295e1
cargo fmt
xgaozoyoe Nov 22, 2023
14f4bdf
remove dryrun service from cli
xgaozoyoe Nov 28, 2023
3d3a7c8
merge dryrun and run by specify dryrun flag
xgaozoyoe Nov 28, 2023
2d02f1a
fix dry_run
junyu0312 Nov 29, 2023
52f2e32
fix tests
xgaozoyoe Nov 29, 2023
3c7560e
add wasm_witness
xgaozoyoe Nov 12, 2023
b8fe209
support standard env in cli
xgaozoyoe Nov 17, 2023
4079e7d
change wasm_witness_inject to wasm_witness_insert
xgaozoyoe Nov 17, 2023
ce1ce95
support wasm_dbg_char
xgaozoyoe Nov 19, 2023
2cec682
move witness api into external foreign host
xgaozoyoe Nov 20, 2023
9383a45
fix: add missing files
xgaozoyoe Nov 20, 2023
4813b11
use better merkle circuit from zkwasm-host-circuits
xgaozoyoe Nov 20, 2023
0aa9a3d
cargo fmt
xgaozoyoe Nov 22, 2023
c2f2fc0
add keccak host support
xgaozoyoe Nov 27, 2023
2f051ad
support keccak
xgaozoyoe Nov 28, 2023
29ca99c
bump zkwasm-host-circuits for zkwasm
xgaozoyoe Nov 29, 2023
c6b6f9b
bump wasmi version
xgaozoyoe Dec 1, 2023
fab06a4
fix uniform circuit verify
xgaozoyoe Dec 4, 2023
1fd9ae8
Merge branch 'main' into host-ops
xgaozoyoe Dec 5, 2023
fa1724d
bump zkwasm-host-circuits
xgaozoyoe Dec 6, 2023
b542532
Merge branch 'host-ops' into host-ops-1.6
xgaozoyoe Dec 6, 2023
541d1c5
bump zkwasm-host-circuits
xgaozoyoe Dec 11, 2023
7047eae
split merkle and datahash
xgaozoyoe Dec 19, 2023
c350117
cleanup merkle host to support hash leaf only
xgaozoyoe Dec 19, 2023
96e0048
support datacache
xgaozoyoe Dec 20, 2023
cb7cc5c
bump zkwasm host circuits to main
xgaozoyoe Dec 21, 2023
c540efc
Merge pull request #220 from DelphinusLab/host-ops
xgaozoyoe Dec 22, 2023
fa0d7ce
get statics for host-ops
xgaozoyoe Dec 25, 2023
ebb92fc
add name for plugin context
junyu0312 Dec 25, 2023
3bf197c
fix tests compilation
xgaozoyoe Dec 25, 2023
88ad294
support tracer in ExecEnv and can be visited in host api
xgaozoyoe Dec 26, 2023
e9d5fbe
polish the host api to get trace count
xgaozoyoe Dec 26, 2023
62b4493
reset keccak256 cursor in keccak_new()
Dec 30, 2023
23873c8
Merge pull request #222 from ethstorage/host-ops-1.6
xgaozoyoe Dec 30, 2023
1d142f2
fix tests
xgaozoyoe Dec 26, 2023
8879d4a
using enum for host mode
xgaozoyoe Dec 31, 2023
3233fc4
feature: support indexed witness in witness host
xgaozoyoe Jan 2, 2024
431795e
bump wasmi version
xgaozoyoe Jan 2, 2024
710750b
remove useless Sequence
xgaozoyoe Jan 2, 2024
716aa39
adjust circuit_with_witness and put println! outside in cli
xgaozoyoe Jan 2, 2024
37f7d68
add comment for picking the first instance data
xgaozoyoe Jan 2, 2024
83c0916
add default implement for ForeignContext
xgaozoyoe Jan 2, 2024
54cdd3e
fix tests
xgaozoyoe Jan 2, 2024
54dd42b
Merge branch 'main' into host-ops
xgaozoyoe Jan 2, 2024
0ec2cc1
cargo fmt code
xgaozoyoe Jan 2, 2024
2a1f88c
deduce the type of Arg for HostEnvBuilder
xgaozoyoe Jan 2, 2024
9f2925f
fix examples
xgaozoyoe Jan 3, 2024
2e8fda9
fix index for witness host api
xgaozoyoe Jan 3, 2024
0e598a6
Merge branch 'main' into host-ops
xgaozoyoe Jan 4, 2024
e14cd42
cargo fmt after merge
xgaozoyoe Jan 4, 2024
37ad4dd
expose witness host context to arg
xgaozoyoe Jan 4, 2024
49f3070
add zkwasm_indexed_witness_push
xgaozoyoe Jan 4, 2024
e5bbb53
use local cache
xgaozoyoe Jan 4, 2024
469eb87
fix counter of merkle
xgaozoyoe Jan 4, 2024
99206c6
Merge pull request #229 from DelphinusLab/host-ops
xgaozoyoe Jan 5, 2024
4c09ac9
bugfix: witness push and insert
xgaozoyoe Jan 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
split merkle and datahash
xgaozoyoe committed Dec 19, 2023

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit 7047eae3c6e4774c6557bc16222b49f62bf5687f
165 changes: 165 additions & 0 deletions crates/host/src/host/merkle_helper/datahash.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,165 @@
use delphinus_zkwasm::runtime::host::host_env::HostEnv;
use delphinus_zkwasm::runtime::host::ForeignContext;
use halo2_proofs::pairing::bn256::Fr;
use std::cell::RefCell;
use std::rc::Rc;
use zkwasm_host_circuits::host::datahash as datahelper;
use zkwasm_host_circuits::host::datahash::DataHashRecord;
use zkwasm_host_circuits::host::db::TreeDB;
use zkwasm_host_circuits::host::ForeignInst::CacheSetMode;
use zkwasm_host_circuits::host::ForeignInst::CacheSetHash;
use zkwasm_host_circuits::host::ForeignInst::CacheFetchData;
use zkwasm_host_circuits::host::ForeignInst::CacheStoreData;
use zkwasm_host_circuits::host::Reduce;
use zkwasm_host_circuits::host::ReduceRule;

const FETCH_MODE: u64 = 0;
const STORE_MODE: u64 = 1;

pub struct CacheContext {
pub mode: u64,
pub hash: Reduce<Fr>,
pub data: Vec<u64>,
pub fetch: bool,
pub mongo_datahash: datahelper::MongoDataHash,
pub tree_db: Option<Rc<RefCell<dyn TreeDB>>>,
}

fn new_reduce(rules: Vec<ReduceRule<Fr>>) -> Reduce<Fr> {
Reduce { cursor: 0, rules }
}

impl CacheContext {
pub fn new(tree_db: Option<Rc<RefCell<dyn TreeDB>>>) -> Self {
CacheContext {
mode: 0,
hash: new_reduce(vec![ReduceRule::Bytes(vec![], 4)]),
fetch: false,
data: vec![],
mongo_datahash: datahelper::MongoDataHash::construct([0; 32], tree_db.clone()),
tree_db,
}
}


pub fn set_mode(&mut self, v: u64) {
self.mode = v;
}

pub fn set_data_hash(&mut self, v: u64) {
self.hash.reduce(v);
if self.hash.cursor == 0 {
let hash = self.hash.rules[0].bytes_value().unwrap().try_into().unwrap();
if v == FETCH_MODE {
let datahashrecord = self.mongo_datahash.get_record(&hash).unwrap();
self.data = datahashrecord.map_or(vec![], |r| {
r.data
.chunks_exact(8)
.into_iter()
.into_iter()
.map(|x| u64::from_le_bytes(x.try_into().unwrap()))
.collect::<Vec<u64>>()
});
self.fetch = false;
} else if v == STORE_MODE {
// put data and hash into mongo_datahash
if !self.data.is_empty() {
self.mongo_datahash
.update_record({
DataHashRecord {
hash: hash.try_into().unwrap(),
data: self
.data
.iter()
.map(|x| x.to_le_bytes())
.flatten()
.collect::<Vec<u8>>(),
}
})
.unwrap();
}
}
}
}

pub fn fetch_data(&mut self) -> u64 {
if self.fetch == false {
self.fetch = true;
self.data.reverse();
self.data.len() as u64
} else {
self.data.pop().unwrap()
}
}

pub fn store_data(&mut self, v: u64) {
self.data.push(v);
}
}

impl CacheContext {}

impl ForeignContext for CacheContext {}

use specs::external_host_call_table::ExternalHostCallSignature;
pub fn register_merkle_foreign(env: &mut HostEnv, tree_db: Option<Rc<RefCell<dyn TreeDB>>>) {
let foreign_merkle_plugin = env
.external_env
.register_plugin("foreign_cache", Box::new(CacheContext::new(tree_db)));

env.external_env.register_function(
"cache_set_mode",
CacheSetMode as usize,
ExternalHostCallSignature::Argument,
foreign_merkle_plugin.clone(),
Rc::new(
|context: &mut dyn ForeignContext, args: wasmi::RuntimeArgs| {
let context = context.downcast_mut::<CacheContext>().unwrap();
context.set_mode(args.nth(0));
None
},
),
);

env.external_env.register_function(
"cache_set_hash",
CacheSetHash as usize,
ExternalHostCallSignature::Argument,
foreign_merkle_plugin.clone(),
Rc::new(
|context: &mut dyn ForeignContext, args: wasmi::RuntimeArgs| {
let context = context.downcast_mut::<CacheContext>().unwrap();
context.set_data_hash(args.nth(0));
None
},
),
);

env.external_env.register_function(
"cache_store_data",
CacheStoreData as usize,
ExternalHostCallSignature::Argument,
foreign_merkle_plugin.clone(),
Rc::new(
|context: &mut dyn ForeignContext, args: wasmi::RuntimeArgs| {
let context = context.downcast_mut::<CacheContext>().unwrap();
context.store_data(args.nth(0));
None
},
),
);

env.external_env.register_function(
"cache_fetch_data",
CacheFetchData as usize,
ExternalHostCallSignature::Return,
foreign_merkle_plugin.clone(),
Rc::new(
|context: &mut dyn ForeignContext, _args: wasmi::RuntimeArgs| {
let context = context.downcast_mut::<CacheContext>().unwrap();
let ret = Some(wasmi::RuntimeValue::I64(context.fetch_data() as i64));
ret
},
),
);
}
114 changes: 6 additions & 108 deletions crates/host/src/host/merkle_helper/merkle.rs
Original file line number Diff line number Diff line change
@@ -25,8 +25,7 @@ pub struct MerkleContext {
pub get_root: Reduce<Fr>,
pub address: Reduce<Fr>,
pub set: Reduce<Fr>,
pub get: Reduce<Fr>,
pub data: Vec<u64>,
pub data: [u64; 4],
pub data_cursor: usize,
pub fetch: bool,
pub mongo_merkle: Option<merklehelper::MongoMerkle<MERKLE_TREE_HEIGHT>>,
@@ -45,14 +44,8 @@ impl MerkleContext {
get_root: new_reduce(vec![ReduceRule::Bytes(vec![], 4)]),
address: new_reduce(vec![ReduceRule::U64(0)]),
set: new_reduce(vec![ReduceRule::Bytes(vec![], 4)]),
get: new_reduce(vec![
ReduceRule::U64(0),
ReduceRule::U64(0),
ReduceRule::U64(0),
ReduceRule::U64(0),
]),
fetch: false,
data: vec![],
data: [0; 4],
data_cursor: 0,
mongo_merkle: None,
mongo_datahash: datahelper::MongoDataHash::construct([0; 32], tree_db.clone()),
@@ -93,7 +86,7 @@ impl MerkleContext {
}

pub fn merkle_address(&mut self, v: u64) {
self.data = vec![];
self.data = [0; 4];
self.fetch = false;
self.address.reduce(v);
}
@@ -110,22 +103,6 @@ impl MerkleContext {
let hash = self.set.rules[0].bytes_value().unwrap();
mt.update_leaf_data_with_proof(index, &hash)
.expect("Unexpected failure: update leaf with proof fail");
// put data and hash into mongo_datahash if the data is binded to the merkle tree leaf
if !self.data.is_empty() {
self.mongo_datahash
.update_record({
DataHashRecord {
hash: hash.try_into().unwrap(),
data: self
.data
.iter()
.map(|x| x.to_le_bytes())
.flatten()
.collect::<Vec<u8>>(),
}
})
.unwrap();
}
}
}

@@ -139,63 +116,11 @@ impl MerkleContext {
let (leaf, _) = mt
.get_leaf_with_proof(index)
.expect("Unexpected failure: get leaf fail");
let cursor = self.get.cursor;
let values = leaf.data_as_u64();
self.get.reduce(values[self.get.cursor]);
// fetch data if we get the target hash
if self.get.cursor == 0 {
let hash: [u8; 32] = vec![
self.get.rules[0]
.u64_value()
.unwrap()
.to_le_bytes()
.to_vec(),
self.get.rules[1]
.u64_value()
.unwrap()
.to_le_bytes()
.to_vec(),
self.get.rules[2]
.u64_value()
.unwrap()
.to_le_bytes()
.to_vec(),
self.get.rules[3]
.u64_value()
.unwrap()
.to_le_bytes()
.to_vec(),
]
.into_iter()
.flatten()
.collect::<Vec<u8>>()
.try_into()
.unwrap();
let datahashrecord = self.mongo_datahash.get_record(&hash).unwrap();
self.data = datahashrecord.map_or(vec![], |r| {
r.data
.chunks_exact(8)
.into_iter()
.into_iter()
.map(|x| u64::from_le_bytes(x.try_into().unwrap()))
.collect::<Vec<u64>>()
});
}
values[cursor]
}

pub fn merkle_fetch_data(&mut self) -> u64 {
if self.fetch == false {
self.fetch = true;
self.data.reverse();
self.data.len() as u64
} else {
self.data.pop().unwrap()
if self.data_cursor == 0 {
self.data = values;
}
}

pub fn merkle_put_data(&mut self, v: u64) {
self.data.push(v);
values[self.data_cursor]
}
}

@@ -236,19 +161,6 @@ pub fn register_merkle_foreign(env: &mut HostEnv, tree_db: Option<Rc<RefCell<dyn
),
);

env.external_env.register_function(
"merkle_fetch_data",
MerkleFetchData as usize,
ExternalHostCallSignature::Return,
foreign_merkle_plugin.clone(),
Rc::new(
|context: &mut dyn ForeignContext, _args: wasmi::RuntimeArgs| {
let context = context.downcast_mut::<MerkleContext>().unwrap();
Some(wasmi::RuntimeValue::I64(context.merkle_fetch_data() as i64))
},
),
);

env.external_env.register_function(
"merkle_address",
MerkleAddress as usize,
@@ -263,20 +175,6 @@ pub fn register_merkle_foreign(env: &mut HostEnv, tree_db: Option<Rc<RefCell<dyn
),
);

env.external_env.register_function(
"merkle_put_data",
MerklePutData as usize,
ExternalHostCallSignature::Argument,
foreign_merkle_plugin.clone(),
Rc::new(
|context: &mut dyn ForeignContext, args: wasmi::RuntimeArgs| {
let context = context.downcast_mut::<MerkleContext>().unwrap();
context.merkle_put_data(args.nth(0));
None
},
),
);

env.external_env.register_function(
"merkle_set",
MerkleSet as usize,
1 change: 1 addition & 0 deletions crates/host/src/host/merkle_helper/mod.rs
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
pub mod merkle;
pub mod datahash;