Skip to content

Commit

Permalink
Containerize tool with automatic file watcher
Browse files Browse the repository at this point in the history
  • Loading branch information
dickwolff authored Feb 2, 2024
2 parents 36a115c + b33e85f commit edb9a7c
Show file tree
Hide file tree
Showing 13 changed files with 325 additions and 82 deletions.
45 changes: 45 additions & 0 deletions .github/workflows/docker.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
name: docker

on:
push:
branches:
- 'main'

jobs:
docker:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
with:
fetch-depth: 0

- name: Install GitVersion
uses: gittools/actions/gitversion/[email protected]
with:
versionSpec: '5.x'

- name: Run GitVersion
uses: gittools/actions/gitversion/[email protected]
with:
useConfigFile: true

- name: Set up QEMU
uses: docker/setup-qemu-action@v3

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Login to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Build and push
uses: docker/build-push-action@v5
with:
push: ${{ github.event_name != 'pull_request' }}
tags: |
dickwolff/export-to-ghostfolio:latest
dickwolff/export-to-ghostfolio:${{ env.GitVersion_MajorMinorPatch }}
File renamed without changes.
13 changes: 13 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
FROM node:20-alpine3.19

WORKDIR /app

COPY . .

RUN npm install

RUN mkdir /var/e2g-input
RUN mkdir /var/e2g-output

ENTRYPOINT [ "npm" ]
CMD ["run", "watch"]
15 changes: 15 additions & 0 deletions GitVersion.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
next-version: 0.3.0
assembly-informational-format: "{NuGetVersion}"
mode: ContinuousDeployment
branches:
master:
regex: main
mode: ContinuousDelivery
tag: ""
increment: Patch
feature:
regex: ^feature?[/-]
mode: ContinuousDelivery
tag: ""
increment: Patch
source-branches: ["main"]
49 changes: 45 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,47 @@ This tool allows you to convert a multiple transaction exports (CSV) to an impor

Is your broker not in the list? Feel free to create an [issue](https://github.com/dickwolff/Export-To-Ghostfolio/issues/new) or, even better, build it yourself and create a [pull request](https://github.com/dickwolff/Export-To-Ghostfolio/compare)!

## System requirements
## How to use

The tool requires you to install the latest LTS version of Node, which you can download [here](https://nodejs.org/en/download/). The tool can run on any OS on which you can install Node.
You can run the tool on your local machine by cloning this repository. You can also run the tool inside a Docker container. See the runtime specific instructions below.

## How to use
## Docker

<details>
<summary>View instructions</summary>

### System requirements

To run the Docker container you need to have [Docker](https://docs.docker.com/get-docker/) installed on your machine. The image is publishe to [Docker Hub](https://hub.docker.com/repository/docker/dickwolff/export-to-ghostfolio/). You can then run the image like:

```
docker run -d -v /C/.../docker_in:/var/e2g-input -v /C/.../docker_out:/var/e2g-output --env GHOSTFOLIO_ACCOUNT_ID=xxxxxxx dickwolff/export-to-ghostfolio
```

The following parameters can be given to the Docker run command.

| Command | Optional | Description |
| ------- | -------- | ----------- |
| ` -v {local_in-folder}:/var/e2g-input` | N | The input folder where you put the files to be processed |
| `-v {local_out_folder}:/var/e2g-output` | N | The output folder where the Ghostfolio import JSON will be placed. Also the input file will be moved here when an error ocurred while processing the file. |
| `--env GHOSTFOLIO_ACCOUNT_ID=xxxxxxx` | N | Your Ghostolio account ID <sup>1</sup> |
| `--env USE_POLLING=true` | Y | When set to true, the container will continously look for new files to process and the container will not stop. |
| `--env DEBUG_LOGGING=true` | Y | When set to true, the container will show logs in more detail, useful for error tracing. |

1: You can retrieve your Ghostfolio account ID by going to Accounts > select your account and copying the ID from the URL.

![image](https://user-images.githubusercontent.com/5620002/203353840-f5db7323-fb2f-4f4f-befc-e4e340466a74.png)

</details>

## Run locally

<details>
<summary>View instructions</summary>

### System requirements

The tool requires you to install the latest LTS version of Node, which you can download [here](https://nodejs.org/en/download/). The tool can run on any OS on which you can install Node.

### Download transaction export

Expand All @@ -41,6 +77,7 @@ Login to your Finpension account. Select your portfolio from the landing page. T
Login to your Swissquote account. From the bar menu click on “Transactions”. Select the desired time period as well as types and then select the “export CSV” button to the right.

#### Schwab

Login to your Schwab account. Go to “Accounts” then “History”. Select the account you want to download details from. Select the “Date Range” and select “Export” (csv). Save the file.

![Export instructions for Schwab](./assets/export-schwab.jpg)
Expand All @@ -60,7 +97,7 @@ The repository contains a sample `.env` file. Rename this from `.env.sample`.
![image](https://user-images.githubusercontent.com/5620002/203353840-f5db7323-fb2f-4f4f-befc-e4e340466a74.png)
- Optionally you can enable debug logging by setting the `DEBUG_LOGGING` variable to `TRUE`.

You can now run `npm run start [exporttype]`. See the table with run commands below. The tool will open your export and will convert this. It retrieves the symbols that are supported with YAHOO Finance (e.g. for European stocks like `ASML`, it will retrieve `ASML.AS` by the corresponding ISIN).
You can now run `npm run start [exporttype]`. See the table with run commands below. The tool will open your export and will convert this. It retrieves the symbols that are supported with YAHOO Finance (e.g. for European stocks like `ASML`, it will retrieve `ASML.AS` by the corresponding ISIN).

| Exporter | Run command |
| ----------- | ----------------------------------- |
Expand All @@ -70,6 +107,10 @@ You can now run `npm run start [exporttype]`. See the table with run commands be
| Swissquote | `run start swissquote` (or `sq`) |
| Schwab | `run start schwab` |

</details>

## Import to Ghostfolio

The export file can now be imported in Ghostfolio by going to Portfolio > Activities and pressing the 3 dots at the top right of the table. Since Ghostfolio 1.221.0, you can now preview the import and validate the data has been converted correctly. When it is to your satisfaction, press import to add the activities to your portfolio.

![image](https://user-images.githubusercontent.com/5620002/203356387-1f42ca31-7cff-44a5-8f6c-84045cf7101e.png)
6 changes: 0 additions & 6 deletions nodemon.json

This file was deleted.

8 changes: 4 additions & 4 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
"name": "export-to-ghostfolio",
"version": "1.0.0",
"description": "Convert multiple broker exports to Ghostfolio import",
"main": "index.js",
"scripts": {
"start": "nodemon",
"start": "ts-node ./src/manual.ts",
"watch": "ts-node ./src/watcher.ts",
"test": "jest --coverage"
},
"author": "Dick Wolff",
Expand All @@ -16,15 +16,15 @@
"@types/jest": "^29.5.11",
"@types/node": "^20.10.4",
"jest": "^29.7.0",
"nodemon": "^3.0.2",
"ts-jest": "^29.1.1",
"ts-node": "^10.9.1",
"typescript": "^5.3.3"
},
"dependencies": {
"@types/cli-progress": "^3.11.5",
"chokidar": "^3.5.3",
"cli-progress": "^3.12.0",
"cross-fetch": "^4.0.0",
"closest-match": "^1.3.3",
"csv-parse": "^5.5.2",
"dayjs": "^1.11.10",
"dotenv": "^16.3.1",
Expand Down
84 changes: 84 additions & 0 deletions src/converter.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
import path from "path";
import * as fs from "fs";
import { GhostfolioExport } from "./models/ghostfolioExport";
import { DeGiroConverter } from "./converters/degiroConverter";
import { SchwabConverter } from "./converters/schwabConverter";
import { DeGiroConverterV2 } from "./converters/degiroConverterV2";
import { AbstractConverter } from "./converters/abstractconverter";
import { Trading212Converter } from "./converters/trading212Converter";
import { SwissquoteConverter } from "./converters/swissquoteConverter";
import { FinpensionConverter } from "./converters/finpensionConverter";

export function createAndRunConverter(converterType: string, inputFilePath: string, outputFilePath: string, completionCallback: CallableFunction, errorCallback: CallableFunction) {

// Verify if Ghostolio account ID is set (because without it there can be no valid output).
if (!process.env.GHOSTFOLIO_ACCOUNT_ID) {
return errorCallback(new Error("Environment variable GHOSTFOLIO_ACCOUNT_ID not set!"));
}

const converterTypeLc = converterType.toLocaleLowerCase();

// Determine convertor type.
const converter = createConverter(converterTypeLc);

// Map the file to a Ghostfolio import.
converter.readAndProcessFile(inputFilePath, (result: GhostfolioExport) => {

console.log("[i] Processing complete, writing to file..")

// Write result to file.
const outputFileName = path.join(outputFilePath, `ghostfolio-${converterTypeLc}.json`);
const fileContents = JSON.stringify(result);
fs.writeFileSync(outputFileName, fileContents, { encoding: "utf-8" });

console.log(`[i] Wrote data to '${outputFileName}.json'!`);

completionCallback();

}, (error) => errorCallback(error));
}

function createConverter(converterType: string): AbstractConverter {

let converter: AbstractConverter;

switch (converterType) {
case "t212":
case "trading212":
console.log("[i] Processing file using Trading212 converter");
converter = new Trading212Converter();
break;
case "degiro":
console.log("[i] Processing file using DeGiro converter");
console.log("[i] NOTE: There is a new version available of the DeGiro converter");
console.log("[i] The new converter has multiple record parsing improvements and also supports platform fees.");
console.log("[i] The new converter is currently in beta and we're looking for your feedback!");
console.log("[i] You can run the beta converter with the command 'npm run start degiro-v2'.");
converter = new DeGiroConverter();
break;
case "degiro-v2":
console.log("[i] Processing file using DeGiro converter (V2 Beta)");
console.log("[i] NOTE: You are running a converter that is currently in beta.");
console.log("[i] If you have any issues, please report them on GitHub. Many thanks!");
converter = new DeGiroConverterV2();
break;
case "fp":
case "finpension":
console.log("[i] Processing file using Finpension converter");
converter = new FinpensionConverter();
break;
case "sq":
case "swissquote":
console.log("[i] Processing file using Swissquote converter");
converter = new SwissquoteConverter();
break;
case "schwab":
console.log("[i] Processing file using Schwab converter");
converter = new SchwabConverter();
break;
default:
throw new Error(`Unknown converter '${converterType}' provided`);
}

return converter;
}
2 changes: 1 addition & 1 deletion src/converters/degiroConverterV2.test.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { DeGiroConverterV2 } from "./degiroConverterV2";

describe("degiroConverter", () => {
describe("degiroConverterV2", () => {

it("should construct", () => {

Expand Down
13 changes: 13 additions & 0 deletions src/converters/schwabConverter.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
import { SchwabConverter } from "./schwabConverter";

describe("SchwabConverter", () => {

it("should construct", () => {

// Act
const sut = new SchwabConverter();

// Asssert
expect(sut).toBeTruthy();
});
});
67 changes: 0 additions & 67 deletions src/index.ts

This file was deleted.

16 changes: 16 additions & 0 deletions src/manual.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
import { createAndRunConverter } from "./converter";

// Check if converter was specified.
if (process.argv.length != 3) {
console.log("[e] Invalid run command: converter not specified!");;
}
else {

require("dotenv").config();

// Define import file path.
const inputFile = process.env.INPUT_FILE;

// Determine convertor type and run conversion.
createAndRunConverter(process.argv[2].toLocaleLowerCase(), inputFile, ".", () => { }, () => { });
}
Loading

0 comments on commit edb9a7c

Please sign in to comment.