[CCK-4] Add plain text files to build process of artifacts #56

Audiosutras · 2020-10-08T19:49:04Z

Fixes

Description

This PR adds the creation of .txt files for licenses' legal code to our build process. By clicking view legal code as plain text link users will download the plain text version of the legal code.

Technical details

Going forward pandoc will need to be installed.

Please click on the link above for more information on installing pandoc.
Linux:

brew install pandoc

Windows:

choco install pandoc

Checklist

My pull request has a descriptive title (not a vague title like Update index.md).
My pull request targets the default branch of the repository (main or master).
My commit messages follow best practices.
My code follows the established code style of the repository.
I added tests for the changes I made (if applicable).
I added or updated documentation (if applicable).
I tried running the project locally and verified that there are no
visible errors.

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

cc_licenses/templates/includes/view_legal_code_link_plain_text.html

licenses/management/commands/publish.py

… cc55-plaintexttemplates

dpoirier

I'm really impressed with how well this is working. I suggested some improvements.

dpoirier · 2020-10-14T12:35:09Z

.travis.yml

+before-install:
+  - sudo apt-get -y install pandoc
+


It looks like Travis has a specific section of the config file that can be used to tell them to add apt packages - https://docs.travis-ci.com/user/installing-dependencies/#adding-apt-packages - it might be better to use that than relying on apt-get.

dpoirier · 2020-10-14T12:35:56Z

.travis.yml

@@ -18,6 +18,9 @@ cache:
 env:
  - DJANGO_SETTINGS_MODULE="cc_licenses.settings.dev"

+before-install:
+  - sudo apt-get -y install pandoc


Can you add pandoc and the version we've tested with to README.rst in the list of dependencies?

dpoirier · 2020-10-14T12:37:42Z

cc_licenses/templates/includes/view_legal_code_link_plain_text.html

+
+<script>
+  document.addEventListener("DOMContentLoaded", function() {
+    const downloadLink = document.querySelector("a#view-plain-text")
+    path = "{{ request.get_full_path }}"
+    if (path.endsWith("legalcode")) {
+      downloadLink.href = "{{ request.get_full_path}}/index.txt"
+    }
+  })
+</script>


It'd be nice if this worked without needing JavaScript. Can we figure out what the plain text link should be in the view and pass it to the template in the context?

dpoirier · 2020-10-14T12:39:07Z

licenses/tests/test_views.py

+            )
+            self.assertEqual(200, rsp.status_code)
+            self.assertGreater(len(rsp.content.decode()), 0)
+


Once we've added the download link to the template context, we might want to add something to the license view tests to check that it's there and correct.

dpoirier · 2020-10-14T12:43:35Z

licenses/views.py

-        )
+        if not is_plain_text:
+            return render(request, **kwargs)
+        response = HttpResponse(content_type='text/plain; charset="utf-8";')


Can you remove the last semicolon? https://www.w3.org/International/articles/http-charset/index

licenses/views.py

dpoirier · 2020-10-19T13:44:07Z

README.rst

+    (cc_licenses)$ brew install [email protected]
+
+or for windows via chocolatey:
+


I think homebrew is for Mac, not Linux. But it might be simpler to just link to the install instructions at https://pandoc.org/installing.html rather than trying to give instructions here.

done and thanks. I should have written apt-get I believe.

dpoirier · 2020-10-19T13:54:33Z

cc_licenses/templates/legalcode_40_plain_text.html

@@ -0,0 +1,331 @@
+{% extends 'base.html' %}


Do we need a separate template for the HTML for plain text? It looks almost identical to the regular one.

I think so if we want to eventually better support rtl/ltr languages. If we want drop this as a requirement for now. I think we can go back to the originally implementation where we use beautifulsoup4 to parse hml = render_to_string(**kwargs). #56 (comment)

what do you think? We also will have to add back id=plain-text-marker to each of the templates we are using for licenses.

dpoirier · 2020-10-19T13:58:18Z

licenses/views.py

+        template_name="legalcode_40_page.html"
+        if not is_plain_text
+        else "legalcode_40_plain_text.html",


Maybe it's just me, but I always find it just a little harder to read and understand a negated condition (e.g. if not ...) and try to rewrite it without the not if there's not some good reason that if not is better in that place. Could you look at the if nots in this PR and see if you think some of them would be a little clearer rewritten without the not?

yes, done. reworked the if statements

dpoirier

Just a couple more comments.

dpoirier · 2020-11-04T15:59:05Z

licenses/urls.py

@@ -152,6 +152,24 @@ def distill_wireframes():
        name="licenses_default_jurisdiction_and_language",
        kwargs=dict(language_code=DEFAULT_LANGUAGE_CODE, jurisdiction=""),
    ),
+    path(
+        # Jurisdiction empty:
+        # e.g. /licenses/by/4.0/legalcode.es - license BY 4.0 Spanish


Can you update the example URL in the comment here and below so they look like the URLs that these path()s define?

yes thanks for catching this

dpoirier · 2020-11-04T16:00:43Z

licenses/views.py

+            "link_to_plain_text_file": f"{ legalcode.license_url() }/index.txt"
+            if legalcode.license_url().endswith("legalcode")
+            else f"{ legalcode.license_url() }.txt",


Could we just define a plaintext_url() method on the LegalCode class that returns the URL we need, like the license_url() and deed_url() methods?

santisoler · 2020-11-09T13:45:31Z

Thanks @Audiosutras for making this possible and thanks @dpoirier for your contributions to the PR!

Sorry for my ignorance, but I couldn't found the view as plain text button on https://creativecommons.org. Could you point me into the right direction?

dpoirier · 2020-11-09T15:29:11Z

I couldn't found the view as plain text button on https://creativecommons.org. Could you point me into the right direction?

Sorry @santisoler if it's not clear. This repository is still a work in progress and has not been deployed to creativecommons.org.

santisoler · 2020-11-09T15:32:41Z

Thanks for the response! I suspected that the latest changes weren't deployed yet, but wasn't sure. I'll wait then for the website to get updated. Meanwhile I'll try to locally serve the version on master to see the changes. Thanks again for the effort!

santisoler · 2021-04-23T20:11:17Z

Hi @dpoirier! Has the code from this PR been deployed to the CC website? Can we grab translated .txt versions of the licenses now?

Thanks again for addressing my issue!

TimidRobot · 2021-04-26T20:53:30Z

Hi @dpoirier! Has the code from this PR been deployed to the CC website? Can we grab translated .txt versions of the licenses now?

Thanks again for addressing my issue!

@santisoler No, this program (and the accompanying data in creativecommons/cc-licenses-data has not yet been brought live / deployed to production). We hope to deploy it soon: 2021 Q2 or Q3.

Furthermore, I do not expect the initial deployment to include generated plaintext files (initial deployment will not have translated plaintext files). The plaintext files must be generated in a completely deterministic way as they are used by projects like licensee/licensee for programmatic license identification. Further development of plaintext file generation is currently blocked by Jinja2 Templates · Issue #102 · creativecommons/cc-licenses.

For reference, the currently supported plaintext files are linked from:

santisoler · 2021-04-27T23:23:51Z

Thanks @TimidRobot for the response!

We hope to deploy it soon: 2021 Q2 or Q3.

I hope to see that release this year then! 🚀

Furthermore, I do not expect the initial deployment to include generated plaintext files (initial deployment will not have translated plaintext files). The plaintext files must be generated in a completely deterministic way as they are used by projects like licensee/licensee for programmatic license identification. Further development of plaintext file generation is currently blocked by Jinja2 Templates · Issue #102 · creativecommons/cc-licenses.

I see. So I think I'll keep maintaining my https://github.com/santisoler/cc-licenses repo then. For some reason it caught attention and it's getting a lot of contributions, like adding other CC licenses I haven't added in the first place and even translations to multiple languages. It's not like we are making a big change, but it seems that people need a simple way to add a plain text LICENSE file with a small Markdown notice. I would love to see my repo being merged into CC so more people can benefit from it by the greater discoverability that CC has over my humble repo.

For reference, the currently supported plaintext files are linked from:

Thanks for those links, I actually used them for gathering the English licenses in plain text.

Thanks again for your work and for the quick response!

TimidRobot · 2021-04-28T14:51:15Z

I see. So I think I'll keep maintaining my https://github.com/santisoler/cc-licenses repo then. For some reason it caught attention and it's getting a lot of contributions, like adding other CC licenses I haven't added in the first place and even translations to multiple languages. It's not like we are making a big change, but it seems that people need a simple way to add a plain text LICENSE file with a small Markdown notice. I would love to see my repo being merged into CC so more people can benefit from it by the greater discoverability that CC has over my humble repo.

It's difficult to predict future software development, but I hope I'm able to add both plaintext and markdown file generation for all 4.0 license translations later this year. I will also be thinking about how we can organize the documentation so that using CC Licenses in GitHub/GitLab/etc. repositories can be highlighted and support multiple languages. All that to say I really appreciate the work you and the other contributors have done!

TimidRobot · 2021-04-28T15:01:33Z

@santisoler I created an issue for any further discussion: Improve support of CC Licensed GitHub repositories · Issue #120

Christopher Dixon added 2 commits October 8, 2020 15:02

outputting text file with just text license's legal code

74fd48f

make view plain text button trigger download

8d00432

Audiosutras requested a review from a team October 8, 2020 19:49

Audiosutras closed this Oct 8, 2020

Audiosutras reopened this Oct 8, 2020

dpoirier changed the title ~~[CC-55] Add plain text files to build process of artifacts~~ [CCK-4] Add plain text files to build process of artifacts Oct 12, 2020

dpoirier reviewed Oct 12, 2020

View reviewed changes

cc_licenses/templates/includes/view_legal_code_link_plain_text.html Outdated Show resolved Hide resolved

licenses/management/commands/publish.py Outdated Show resolved Hide resolved

Christopher Dixon added 5 commits October 13, 2020 11:08

Merge branch 'develop' of github.com:creativecommons/cc-licenses into…

51afa26

… cc55-plaintexttemplates

use content_type text/plain and utf-8 charset

bbaefbc

fix content_type so HTTPResponse charset is utf-8

3531dd6

add test for plain_text files

1937d67

include pandoc in travis build

1b86c41

dpoirier reviewed Oct 14, 2020

View reviewed changes

Christopher Dixon added 3 commits October 14, 2020 14:59

revisions based on comments other than plain text files rtl/ltr support.

a0c9ba3

pass full html doc to pandoc

91ab8f9

fix travis install error

d7359d5

dpoirier reviewed Oct 26, 2020

View reviewed changes

Christopher Dixon added 3 commits October 29, 2020 16:18

revisions based on Dan's review

d5b1635

merge with develop, remove plain-text template and use bs4 to render

7ddc8ad

black

2be2f4e

dpoirier reviewed Nov 4, 2020

View reviewed changes

Christopher Dixon added 2 commits November 4, 2020 20:36

remove plain_text_link in test_views.

e09bd14

update comments for plain text url paths

8198c6b

dpoirier approved these changes Nov 5, 2020

View reviewed changes

Merge branch 'develop' into cc55-plaintexttemplates

1093a21

Audiosutras merged commit 1753213 into develop Nov 5, 2020

Audiosutras deleted the cc55-plaintexttemplates branch November 5, 2020 18:26

santisoler mentioned this pull request Apr 23, 2021

Add Italian translation santisoler/cc-licenses#13

Closed

4 tasks

creativecommons locked as resolved and limited conversation to collaborators Apr 28, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CCK-4] Add plain text files to build process of artifacts #56

[CCK-4] Add plain text files to build process of artifacts #56

Audiosutras commented Oct 8, 2020

dpoirier left a comment

dpoirier Oct 14, 2020

Audiosutras Oct 14, 2020

dpoirier Oct 14, 2020

Audiosutras Oct 14, 2020

dpoirier Oct 14, 2020

Audiosutras Oct 14, 2020

dpoirier Oct 14, 2020

Audiosutras Oct 14, 2020

dpoirier Oct 14, 2020

Audiosutras Oct 14, 2020

dpoirier Oct 19, 2020

Audiosutras Oct 29, 2020

dpoirier Oct 19, 2020

Audiosutras Oct 29, 2020

Audiosutras Oct 29, 2020

dpoirier Oct 19, 2020

Audiosutras Oct 29, 2020

dpoirier left a comment

dpoirier Nov 4, 2020

Audiosutras Nov 5, 2020

Audiosutras Nov 5, 2020

dpoirier Nov 4, 2020

Audiosutras Nov 5, 2020

santisoler commented Nov 9, 2020

dpoirier commented Nov 9, 2020

santisoler commented Nov 9, 2020

santisoler commented Apr 23, 2021

TimidRobot commented Apr 26, 2021

santisoler commented Apr 27, 2021

TimidRobot commented Apr 28, 2021

TimidRobot commented Apr 28, 2021

		(cc_licenses)$ brew install [email protected]

		or for windows via chocolatey:

[CCK-4] Add plain text files to build process of artifacts #56

[CCK-4] Add plain text files to build process of artifacts #56

Conversation

Audiosutras commented Oct 8, 2020

Fixes

Description

Technical details

Checklist

Developer Certificate of Origin

dpoirier left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dpoirier left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

santisoler commented Nov 9, 2020

dpoirier commented Nov 9, 2020

santisoler commented Nov 9, 2020

santisoler commented Apr 23, 2021

TimidRobot commented Apr 26, 2021

santisoler commented Apr 27, 2021

TimidRobot commented Apr 28, 2021

TimidRobot commented Apr 28, 2021