Rework documentation

Enforce a more consistent style on the docs. Also correct a number of
typos and stale information.

CONTRIBUTING.md

@@ -0,0 +1,49 @@
+# Maintainer and contributor instructions
+
+## Compiling from source
+
+To compile from source, you'll have to initialize the submodule containing
+librdkafka:
+
+```bash
+git submodule update --init
+```
+
+and then compile using `cargo`, selecting the features that you want.
+Example:
+
+```bash
+cargo build --features "ssl gssapi"
+```
+
+## Tests
+
+### Unit tests
+
+The unit tests can run without a Kafka broker present:
+
+```bash
+cargo test --lib
+```
+
+### Automatic testing
+
+rust-rdkafka contains a suite of tests which is automatically executed by travis in
+docker-compose. Given the interaction with C code that rust-rdkafka has to do, tests
+are executed in valgrind to check eventual memory errors and leaks.
+
+To run the full suite using docker-compose:
+
+```bash
+./test_suite.sh
+```
+
+To run locally, instead:
+
+```bash
+KAFKA_HOST="kafka_server:9092" cargo test
+```
+
+In this case there is a broker expected to be running on `KAFKA_HOST`.
+The broker must be configured with default partition number 3 and topic
+autocreation in order for the tests to succeed.

generate_readme.py

@@ -27,10 +27,14 @@ def parse_template_file(path):
                            if line.startswith(INCLUDE_MARKER)][0]
     except IndexError:
         raise Exception("Missing include marker")
-    include_info = content[marker_position].split('$')
+    include_info = content[marker_position].strip().split('$')
+    doc_path = include_info[1]
+    start = None
+    if len(include_info) > 2:
+        start = include_info[2]
     return Template(
         header=content[0:marker_position], footer=content[marker_position+1:],
-        doc_path=include_info[1], start=include_info[2],
+        doc_path=include_info[1], start=start,
     )
 
 
@@ -42,10 +46,11 @@ def parse_template_file(path):
 for line in template.header:
     output.write(line)
 
-for line in doc:
-    if line.startswith(template.start):
-        output.write(line)
-        break
+if template.start:
+    for line in doc:
+        if line.startswith(template.start):
+            output.write(line)
+            break
 
 for line in doc:
     output.write(line)

rdkafka-sys/CONTRIBUTING.md

@@ -0,0 +1,17 @@
+# Maintainer and contributor instructions
+
+## Bindings
+
+To regenerate the bindings:
+
+``` bash
+git submodule update --init
+cargo install bindgen
+./update-bindings.sh
+```
+
+## Updating
+
+To upgrade change the git submodule in `librdkafka`, check if new errors need to
+be added to `helpers::primive_to_rd_kafka_resp_err_t` and update the version in
+`Cargo.toml`.

rdkafka-sys/README.md

@@ -1,94 +1,86 @@
 # rdkafka-sys
 
-Low level bindings to [librdkafka](https://github.com/edenhill/librdkafka).
+Low level bindings to [librdkafka](https://github.com/edenhill/librdkafka),
+a C library for the [Apache Kafka] protocol with producer, consumer, and
+admin clients.
 
-## Bindings
-
-To regenerate the bindings:
-
-``` bash
-git submodule update --init
-cargo install bindgen
-./update-bindings.sh
-```
+For a safe wrapper, see the [rdkafka] crate.
 
 ## Version
 
-The rdkafka-sys version number is in the format `X.Y.Z+RX.RY.RZ`, where `X.Y.Z`
-is the version of this crate and follows SemVer conventions, while `RX.RY.RZ`
-is the version of the bundled librdkafka.
+The rdkafka-sys version number is in the format `X.Y.Z+RX.RY.RZ`, where
+`X.Y.Z` is the version of this crate and follows SemVer conventions, while
+`RX.RY.RZ` is the version of the bundled librdkafka.
 
 Note that versions before v2.0.0+1.4.2 did not follow this convention, and
-instead directly corresponded to the bundled librdkafka version.
+instead directly correspond to the bundled librdkafka version.
 
 ## Build
 
 ### Known issues
 
 * When any of librdkafka's optional dependencies are enabled, like libz or
   OpenSSL, if you have multiple versions of that library installed upon your
-  system, librdkafka's build system may disagree with Cargo about which version
-  of the library to use! **This can result in subtly broken builds,** if
-  librdkafka compiles against the headers for one version but Cargo links
-  against a different version.  For complete confidence when building release
-  binaries, use an environment like a Docker container or a chroot jail where
-  you can guarantee that only one version of each dependency is present.
-  Unfortunately, the current design of Cargo makes this nearly impossible to
-  fix.
-
-* librdkafka's default build system, which uses a bespoke tool called mklove, is
-  somewhat unreliable, as enabling optional features is best-effort. If a
-  required dependency for an optional feature is not found, the feature will be
-  silently disabled ([details][mklove-bug]). **Using the CMake based build
-  system is strongly encouraged**, if you can take the dependency on CMake.
+  system, librdkafka's build system may disagree with Cargo about which
+  version of the library to use! **This can result in subtly broken
+  builds,** if librdkafka compiles against the headers for one version but
+  Cargo links against a different version.  For complete confidence when
+  building release binaries, use an environment like a Docker container or a
+  chroot jail where you can guarantee that only one version of each
+  dependency is present. The current design of Cargo unfortunately makes
+  this nearly impossible to fix.
 
 ### Features
 
-By default a submodule with the librdkafka sources pinned to a specific commit
-will be used to compile and statically link the library.
+By default a submodule with the librdkafka sources will be used to compile
+and statically link the library.
 
 The **`dynamic-linking`** feature can be used to link rdkafka to a locally
 installed version of librdkafka: if the feature is enabled, the build script
 will use `pkg-config` to check the version of the library installed in the
 system, and it will configure the compiler to dynamically link against it.
-
-The **`cmake-build`** feature builds librdkafka with its [CMake] build system,
-rather than its default [mklove]-based build system. This feature requires that
-CMake is installed on the build machine.
-
-The following features directly correspond to librdkafka features (i.e., flags
-you would pass to `configure` if you were compiling manually).
-
-  * The **`ssl`** feature enables SSL support. By default, the system's OpenSSL
-    library is dynamically linked, but static linking of the version bundled
-    with the openssl-sys crate can be requested with the `ssl-vendored` feature.
-  * The **`gssapi`** feature enables SASL GSSAPI support with Cyrus libsasl2.
-    By default the system's libsasl2 is dynamically linked, but static linking
-    of the version bundled with the sasl2-sys crate can be requested with the
-    `gssapi-vendored` feature.
+The system version of librdkafka must exactly match the version of
+librdkafka bundled with this crate.
+
+The **`cmake-build`** feature builds librdkafka with its [CMake] build
+system, rather than its default [mklove]-based build system. This feature
+requires that CMake is installed on the build machine.
+
+The following features directly correspond to librdkafka features (i.e.,
+flags you would pass to `configure` if you were compiling manually).
+
+  * The **`ssl`** feature enables SSL support. By default, the system's
+    OpenSSL library is dynamically linked, but static linking of the version
+    bundled with the [openssl-sys] crate can be requested with the
+    `ssl-vendored` feature.
+  * The **`gssapi`** feature enables SASL GSSAPI support with Cyrus
+    libsasl2. By default the system's libsasl2 is dynamically linked, but
+    static linking of the version bundled with the [sasl2-sys] crate can be
+    requested with the `gssapi-vendored` feature.
   * The **`libz`** feature enables support for zlib compression. This
-    feature is enabled by default. By default, the system's libz is dynamically
-    linked, but static linking of the version bundled with the libz-sys crate
-    can be requested with the `libz-static` feature.
+    feature is enabled by default. By default, the system's libz is
+    dynamically linked, but static linking of the version bundled with the
+    [libz-sys] crate can be requested with the `libz-static` feature.
   * The **`zstd`** feature enables support for ZSTD compression. By default,
-    this builds and statically links the version bundled with the zstd-sys
-    crate, but dynamic linking of the system's version can be requested with the
-    `zstd-pkg-config` feature.
-  * The **`external-lz4`** feature statically links against the copy of liblz4
-    bundled with the lz4-sys crate. By default, librdkafka statically links
-    against its own bundled version of liblz4. Due to limitations with lz4-sys,
-    it is not yet possible to dynamically link against the system's version of
-    liblz4.
+    this builds and statically links the version bundled with the [zstd-sys]
+    crate, but dynamic linking of the system's version can be requested with
+    the `zstd-pkg-config` feature.
+  * The **`external-lz4`** feature statically links against the copy of
+    liblz4 bundled with the [lz4-sys] crate. By default, librdkafka
+    statically links against its own bundled version of liblz4. Due to
+    limitations with lz4-sys, it is not yet possible to dynamically link
+    against the system's version of liblz4.
 
 All features are disabled by default unless noted otherwise above. The build
-process is defined in [`build.rs`](build.rs).
-
-## Updating
-
-To upgrade change the git submodule in `librdkafka`, check if new errors
-need to be added to `helpers::primive_to_rd_kafka_resp_err_t` and update
-the version in `Cargo.toml`.
+process is defined in [`build.rs`].
 
+[`build.rs`]: https://github.com/fede1024/rust-rdkafka/tree/master/rdkafka-sys/build.rs
+[Apache Kafka]: https://kafka.apache.org
 [CMake]: https://cmake.org
+[libz-sys]: https://crates.io/crates/libz-sys
+[lz4-sys]: https://crates.io/crates/lz4-sys
 [mklove]: https://github.com/edenhill/mklove
-[mklove-bug]: https://github.com/edenhill/librdkafka/pull/2640
+[openssl-sys]: https://crates.io/crates/openssl-sys
+[rdkafka]: https://docs.rs/rdkafka
+[sasl2-sys]: https://docs.rs/sasl2-sys
+[zstd-sys]: https://crates.io/crates/zstd-sys

rdkafka-sys/readme_template

@@ -1 +1,3 @@
-__INCLUDE_RUST_DOC__$src/lib.rs$# rdkafka-sys
+# rdkafka-sys
+
+__INCLUDE_RUST_DOC__$src/lib.rs

rdkafka-sys/src/helpers.rs

@@ -1,3 +1,5 @@
+//! Utility functions.
+
 use crate::types::RDKafkaError;
 use crate::types::RDKafkaError::*;
 use crate::types::RDKafkaRespErr;