Skip to content

Commit

Permalink
added support for JSON array as input
Browse files Browse the repository at this point in the history
fixup! added support for JSON array as input
  • Loading branch information
dcmoura committed Mar 21, 2022
1 parent 25030e7 commit d8c00da
Show file tree
Hide file tree
Showing 3 changed files with 36 additions and 11 deletions.
25 changes: 17 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@ plt "hist(x,30)" < sample.json

![](https://github.com/dcmoura/matplotcli/raw/main/img/hist_sample.png)


The format of the input data format is [JSON lines](https://jsonlines.org), where each line is a valid JSON object. Look at the recipes section to learn how to handle other formats like CSV.
MatplotCLI accepts both [JSON lines](https://jsonlines.org) and arrays of JSON objects as input.
Look at the recipes section to learn how to handle other formats like CSV.

MatplotCLI executes python code (passed as argument) where some handy imports are already done (e.g. `from matplotlib.pyplot import *`) and where the input JSON data is already parsed and available in variables, making plotting easy. Please refer to `matplotlib.pyplot`'s [reference](https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.html#module-matplotlib.pyplot) and [tutorial](https://matplotlib.org/stable/tutorials/introductory/pyplot.html) for comprehensive documentation about the plotting commands.

Expand Down Expand Up @@ -75,17 +75,26 @@ pip install matplotcli

## Recipes and Examples

### Plotting from a json array
### Plotting JSON data

MatplotCLI natively supports JSON lines:

```sh
echo '
{"a":0, "b":1}
{"a":1, "b":0}
{"a":3, "b":3}' |
plt "plot(a,b)"
```

and arrays of JSON objects:

[jq](https://stedolan.github.io/jq/) is a very handy utility whenever we need to handle different JSON flavors. The `-c` option guarantees one JSON object per line in the output.

```sh
echo '[
{"a":0, "b":1},
{"a":1, "b":0},
{"a":3, "b":3}
]' |
jq -c .[] |
{"a":3, "b":3}]' |
plt "plot(a,b)"
```

Expand Down Expand Up @@ -115,7 +124,7 @@ cat file.toml | tomlq -c | plt "plot(x,y)"

### Plotting from a parquet file

`parquet-tools` allows dumping a parquet file to JSON format. `jq -c` makes sure that the output has 1 JSON object per line before handing over to MatplotCLI.
`parquet-tools` allows dumping a parquet file to JSON format. [`jq -c`](https://stedolan.github.io/jq/) makes sure that the output has 1 JSON object per line before handing over to MatplotCLI.

```sh
parquet-tools cat --json my.parquet | jq -c | plt "plot(x,y)"
Expand Down
20 changes: 18 additions & 2 deletions matplotcli.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,24 @@ def make_str_valid_varname(s):


def read_data(vars):
"""reads json lines from stdin"""
lines = [json.loads(line) for line in sys.stdin]
"""reads json lines or an array of jsons from stdin"""
input_str = sys.stdin.read().lstrip() # assuming we won't be loading GB files...
lines = []
if input_str:
first_char = input_str[0]
if first_char == "{":
sys.stderr.write("plt input is JSON lines\n")
lines = [json.loads(line) for line in input_str.splitlines()]
elif first_char == "[":
sys.stderr.write("plt input is a JSON array\n")
lines = json.loads(input_str)
else:
raise Exception(
"plt unable to determine input format, unexpected char '{}'.\n".format(
first_char
)
)

col_names = {key for line in lines for key in line.keys()}
data = {
make_str_valid_varname(col): [line.get(col) for line in lines]
Expand Down
2 changes: 1 addition & 1 deletion setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,5 +42,5 @@
project_urls={
"Source": "https://github.com/dcmoura/matplotcli",
},
version="0.1.0-3",
version="0.2.0",
)

0 comments on commit d8c00da

Please sign in to comment.