Merge from master

README.md

@@ -42,3 +42,10 @@ OPTIONS
 | --usecache  | false         | If true, accept cached results (if available), else force live scan |
 | --grade     | false         | Output only the hostname: grade |
 | --hostcheck | false         | If true, host resolution failure will result in a fatal error |
+
+##Docker
+
+Docker images for this project are available at:
+
+* [https://github.com/jumanjihouse/docker-ssllabs-scan]
+  (https://github.com/jumanjihouse/docker-ssllabs-scan)

ssllabs-api-docs.md

@@ -1,6 +1,6 @@
-# SSL Labs API Documentation v1.16.14 #
+# SSL Labs API Documentation v1.19.27 #
 
-**Last update:** 27 April 2015<br>
+**Last update:** 10 July 2015<br>
 **Author:** Ivan Ristic <[email protected]>
 
 This document explains the SSL Labs Assessment APIs, which can be used to test SSL servers available on the public Internet.
@@ -107,12 +107,12 @@ The field value references the API parameter name that has an incorrect value. T
 The following status codes are used:
 
 * 400 - invocation error (e.g., invalid parameters)
-* 429 - client request rate too high
+* 429 - client request rate too high or too many new assessments too fast
 * 500 - internal error
 * 503 - the service is not available (e.g., down for maintenance)
 * 529 - the service is overloaded
 
-If you get 429, 503, 529, you should sleep for several minutes (e.g., 5, 15, 30 minutes, respectively) then try again. If you're writing an API client tool and get a 529 response, randomize the back-off time. If you get 500, it's best to give up.
+A well-written client should never get a 429 response. If you do get one, it means that you're either submitting new assessments at a rate that is too fast, or that you're not correctly tracking how many concurrent requests you're allowed to have. If you get a 503 or 529 status code, you should sleep for several minutes (e.g., 15 and 30 minutes, respectively) then try again. It's best to randomize the delay, especially if you're writing a client tool -- you don't want everyone to retry exactly at the same time. If you get 500, it means that there's a severe problem with the SSL Labs application itself. A sensible approach would be to mark that one assessment as flawed, but to continue on. However, if you continue to receive 500 responses, it's best to give up.
 
 ### Access Rate and Rate Limiting ###
 
@@ -121,9 +121,9 @@ Please note the following:
 * Server assessments usually take at least 60 seconds. (They are intentionally slow, to avoid harming servers.) Thus, there is no need to poll for the results very often. In fact, polling too often slows down the service for everyone. It's best to use variable polling: 5 seconds until an assessment gets under way (status changes to IN_PROGRESS), then 10 seconds until it completes.
 * Keep down the number of concurrent assessments to a minimum. If you're not in a hurry, test only one hostname at a time.
 
-We may limit your usage of the API, by enforcing a limit on concurrent assessments, and the overall number of assessments performed in a time period. If that happens, we will respond with 429 (Too Many Requests) to API calls that wish to initiate new assessments. Your ability to follow previously initiated assessments, or retrieve assessment results from the cache, will not be impacted. If you receive a 429 response, reduce the number of concurrent assessments.
+We may limit your usage of the API, by enforcing a limit on concurrent assessments, and the overall number of assessments performed in a time period. If that happens, we will respond with 429 (Too Many Requests) to API calls that wish to initiate new assessments. Your ability to follow previously initiated assessments, or retrieve assessment results from the cache, will not be impacted. If you receive a 429 response, reduce the number of concurrent assessments and check that you're not submitting new assessments at a rate higher than allowed.
 
-If the server is overloaded (a condition that is not a result of the client's behaviour), the 529 status code will be used instead. This is not a situation we wish to be in. If you encounter it, take a break and come back after at least 30 minutes of sleep.
+If the server is overloaded (a condition that is not a result of the client's behaviour), the 529 status code will be used instead. This is not a situation we wish to be in. If you encounter it, take a break and come back later.
 
 All successful API calls contain response headers `X-Max-Assessments` and `X-Current-Assessments`. They can be used to calculate how many new
 assessments can be submitted. It is recommended that clients update their internal state after each complete response.
@@ -156,7 +156,7 @@ The remainder of the document explains the structure of the returned objects. Th
 * **criteriaVersion** - grading criteria version (e.g., "2009")
 * **cacheExpiryTime** - when will the assessment results expire from the cache (typically set only for assessment with errors; otherwise the results stay in the cache for as long as there's sufficient room)
 * **endpoints[]** - list of [Endpoint objects](#endpoint)
-* **certHostnames[]** - the list of certificate hostnames collected from the certificates seen during assessment. The hostnames may not be valid.
+* **certHostnames[]** - the list of certificate hostnames collected from the certificates seen during assessment. The hostnames may not be valid. This field is available only if the server certificate doesn't match the requested hostname. In that case, this field saves you some time as you don't have to inspect the certificates yourself to find out what valid hostnames might be.
 
 ### Endpoint ###
 
@@ -243,16 +243,23 @@ The remainder of the document explains the structure of the returned objects. Th
   * bit 0 (1) - SCT in certificate
   * bit 1 (2) - SCT in the stapled OCSP response
   * bit 2 (4) - SCT in the TLS extension (ServerHello)
+* **dhPrimes[]** - list of hex-encoded DH primes used by the server
+* **dhUsesKnownPrimes** - whether the server uses known DH primes:
+  * 0 - no
+  * 1 - yes, but they're not weak
+  * 2 - yes and they're weak
+* **dhYsReuse** - true if the DH ephemeral server value is reused.
+* **logjam** - true if the server uses DH parameters weaker than 1024 bits.
 
 ### Info ###
 
 * **version** - SSL Labs software version as a string (e.g., "1.11.14")
 * **criteriaVersion** - rating criteria version as a string (e.g., "2009f")
 * **maxAssessments** - the maximum number of concurrent assessments the client is allowed to initiate.
 * **currentAssessments** - the number of ongoing assessments submitted by this client.
+* **newAssessmentCoolOff** - the cool-off period after each new assessment; you're not allowed to submit a new assessment before the cool-off expires, otherwise you'll get a 429.
 * **messages** - a list of messages (strings). Messages can be public (sent to everyone) and private (sent only to the invoking client).
                  Private messages are prefixed with "[Private]".
-* **clientMaxAssessments** - deprecated and scheduled for removal in the next release.
 
 ### Key ###
 
@@ -416,3 +423,10 @@ The remainder of the document explains the structure of the returned objects. Th
 * Added Cert.crlRevocationStatus and Cert.ocspRevocationStatus.
 * Added ChainCert.revocationStatus, ChainCert.crlRevocationStatus and ChainCert.ocspRevocationStatus.
 * Added Endpoint.gradeTrustIgnored.
+
+### 1.19.x (Not released) ###
+
+* New EndpointDetails fields: dhPrimes, dhUsesKnownPrimes, dhYsReuse, and logjam.
+* New Info field: newAssessmentCoolOff. There is now a mandatory cool-off period after each new assessment.
+
+

ssllabs-scan.go

@@ -20,6 +20,7 @@
 package main
 
 import "crypto/tls"
+import "errors"
 import "encoding/json"
 import "flag"
 import "fmt"
@@ -50,18 +51,25 @@ const (
 	LOG_TRACE    = 8
 )
 
-var USER_AGENT = "ssllabs-scan v1.1.0 ($Id$)"
+var USER_AGENT = "ssllabs-scan v1.2.0 (dev $Id$)"
 
 var logLevel = LOG_NOTICE
 
+// How many assessment do we have in progress?
 var activeAssessments = 0
 
+// How many assessments does the server think we have in progress?
+var currentAssessments = -1
+
+// The maximum number of assessments we can have in progress at any one time.
 var maxAssessments = -1
 
 var requestCounter uint64 = 0
 
 var apiLocation = "https://api.ssllabs.com/api/v2"
 
+var globalNewAssessmentCoolOff int64 = 1100
+
 var globalIgnoreMismatch = false
 
 var globalStartNew = true
@@ -230,6 +238,10 @@ type LabsEndpointDetails struct {
 	FallbackScsv                   bool
 	Freak                          bool
 	HasSct                         int
+	DhPrimes                       []string
+	DhUsesKnownPrimes              int
+	DhYsReuse                      bool
+	Logjam                         bool
 }
 
 type LabsEndpoint struct {
@@ -271,11 +283,12 @@ type LabsResults struct {
 }
 
 type LabsInfo struct {
-	EngineVersion      string
-	CriteriaVersion    string
-	MaxAssessments     int
-	CurrentAssessments int
-	Messages           []string
+	EngineVersion        string
+	CriteriaVersion      string
+	MaxAssessments       int
+	CurrentAssessments   int
+	NewAssessmentCoolOff int64
+	Messages             []string
 }
 
 func invokeGetRepeatedly(url string) (*http.Response, []byte, error) {
@@ -319,17 +332,41 @@ func invokeGetRepeatedly(url string) (*http.Response, []byte, error) {
 				}
 			}
 
-			// Adjust maximum concurrent requests.
+			// Update current assessments.
+
+			headerValue := resp.Header.Get("X-Current-Assessments")
+			if headerValue != "" {
+				i, err := strconv.Atoi(headerValue)
+				if err == nil {
+					if currentAssessments != i {
+						currentAssessments = i
+
+						if logLevel >= LOG_DEBUG {
+							log.Printf("[DEBUG] Server set current assessments to %v", headerValue)
+						}
+					}
+				} else {
+					if logLevel >= LOG_WARNING {
+						log.Printf("[WARNING] Ignoring invalid X-Current-Assessments value (%v): %v", headerValue, err)
+					}
+				}
+			}
+
+			// Update maximum assessments.
 
-			headerValue := resp.Header.Get("X-Max-Assessments")
+			headerValue = resp.Header.Get("X-Max-Assessments")
 			if headerValue != "" {
 				i, err := strconv.Atoi(headerValue)
 				if err == nil {
 					if maxAssessments != i {
 						maxAssessments = i
 
+						if maxAssessments <= 0 {
+							log.Fatalf("[ERROR] Server doesn't allow further API requests")
+						}
+
 						if logLevel >= LOG_DEBUG {
-							log.Printf("[DEBUG] Server set max concurrent assessments to %v", headerValue)
+							log.Printf("[DEBUG] Server set maximum assessments to %v", headerValue)
 						}
 					}
 				} else {
@@ -354,7 +391,7 @@ func invokeGetRepeatedly(url string) (*http.Response, []byte, error) {
 
 			return resp, body, nil
 		} else {
-			if err.Error() == "EOF" {
+			if strings.Contains(err.Error(), "EOF") {
 				// Server closed a persistent connection on us, which
 				// Go doesn't seem to be handling well. So we'll try one
 				// more time.
@@ -386,11 +423,7 @@ func invokeApi(command string) (*http.Response, []byte, error) {
 		// Status codes 429, 503, and 529 essentially mean try later. Thus,
 		// if we encounter them, we sleep for a while and try again.
 		if resp.StatusCode == 429 {
-			if logLevel >= LOG_NOTICE {
-				log.Printf("[NOTICE] Sleeping for 30 seconds after a %v response", resp.StatusCode)
-			}
-
-			time.Sleep(30 * time.Second)
+			return resp, body, errors.New("Assessment failed: 429")
 		} else if (resp.StatusCode == 503) || (resp.StatusCode == 529) {
 			// In case of the overloaded server, randomize the sleep time so
 			// that some clients reconnect earlier and some later.
@@ -483,6 +516,7 @@ type Event struct {
 }
 
 const (
+	ASSESSMENT_FAILED   = -1
 	ASSESSMENT_STARTING = 0
 	ASSESSMENT_COMPLETE = 1
 )
@@ -497,15 +531,23 @@ func NewAssessment(host string, eventChannel chan Event) {
 	for {
 		myResponse, err := invokeAnalyze(host, startNew, globalFromCache)
 		if err != nil {
-			log.Fatalf("[ERROR] API invocation failed: %v", err)
+			eventChannel <- Event{host, ASSESSMENT_FAILED, nil}
+			return
 		}
 
 		if startTime == -1 {
 			startTime = myResponse.StartTime
 			startNew = false
 		} else {
-			if myResponse.StartTime != startTime {
-				log.Fatalf("[ERROR] Inconsistent startTime. Expected %v, got %v.", startTime, myResponse.StartTime)
+			// Abort this assessment if the time we receive in a follow-up check
+			// is older than the time we got when we started the request. The
+			// upstream code should then retry the hostname in order to get
+			// consistent results.
+			if myResponse.StartTime > startTime {
+				eventChannel <- Event{host, ASSESSMENT_FAILED, nil}
+				return
+			} else {
+				startTime = myResponse.StartTime
 			}
 		}
 
@@ -521,23 +563,30 @@ func NewAssessment(host string, eventChannel chan Event) {
 }
 
 type HostProvider struct {
-	hostnames []string
-	i         int
+	hostnames   []string
+	StartingLen int
 }
 
 func NewHostProvider(hs []string) *HostProvider {
-	hostProvider := HostProvider{hs, 0}
+	hostnames := make([]string, len(hs))
+	copy(hostnames, hs)
+	hostProvider := HostProvider{hostnames, len(hs)}
 	return &hostProvider
 }
 
 func (hp *HostProvider) next() (string, bool) {
-	if hp.i < len(hp.hostnames) {
-		host := hp.hostnames[hp.i]
-		hp.i = hp.i + 1
-		return host, true
-	} else {
+	if len(hp.hostnames) == 0 {
 		return "", false
 	}
+
+	var e string
+	e, hp.hostnames = hp.hostnames[0], hp.hostnames[1:]
+
+	return e, true
+}
+
+func (hp *HostProvider) retry(host string) {
+	hp.hostnames = append(hp.hostnames, host)
 }
 
 type Manager struct {
@@ -568,7 +617,7 @@ func (manager *Manager) startAssessment(h string) {
 func (manager *Manager) run() {
 	transport := &http.Transport{
 		TLSClientConfig:   &tls.Config{InsecureSkipVerify: globalInsecure},
-		DisableKeepAlives: true,
+		DisableKeepAlives: false,
 		Proxy:             http.ProxyFromEnvironment,
 	}
 
@@ -604,10 +653,23 @@ func (manager *Manager) run() {
 
 	moreAssessments := true
 
+	if labsInfo.NewAssessmentCoolOff >= 1000 {
+		globalNewAssessmentCoolOff = 100 + labsInfo.NewAssessmentCoolOff
+	} else {
+		if logLevel >= LOG_WARNING {
+			log.Printf("[WARNING] Info.NewAssessmentCoolOff too small: %v", labsInfo.NewAssessmentCoolOff)
+		}
+	}
+
 	for {
 		select {
 		// Handle assessment events (e.g., starting and finishing).
 		case e := <-manager.BackendEventChannel:
+			if e.eventType == ASSESSMENT_FAILED {
+				activeAssessments--
+				manager.hostProvider.retry(e.host)
+			}
+
 			if e.eventType == ASSESSMENT_STARTING {
 				if logLevel >= LOG_INFO {
 					log.Printf("[INFO] Assessment starting: %v", e.host)
@@ -618,7 +680,6 @@ func (manager *Manager) run() {
 				if logLevel >= LOG_INFO {
 					msg := ""
 
-					// Missing C's ternary operator here.
 					if len(e.report.Endpoints) == 0 {
 						msg = fmt.Sprintf("[WARN] Assessment failed: %v (%v)", e.host, e.report.StatusMessage)
 					} else if len(e.report.Endpoints) > 1 {
@@ -648,22 +709,25 @@ func (manager *Manager) run() {
 				if logLevel >= LOG_DEBUG {
 					log.Printf("[DEBUG] Active assessments: %v (more: %v)", activeAssessments, moreAssessments)
 				}
+			}
 
-				// Are we done?
-				if (activeAssessments == 0) && (moreAssessments == false) {
-					close(manager.FrontendEventChannel)
-					return
-				}
+			// Are we done?
+			if (activeAssessments == 0) && (moreAssessments == false) {
+				close(manager.FrontendEventChannel)
+				return
 			}
 
 			break
 
 		// Once a second, start a new assessment, provided there are
 		// hostnames left and we're not over the concurrent assessment limit.
 		default:
-			<-time.NewTimer(time.Second).C
+			if manager.hostProvider.StartingLen > 0 {
+				<-time.NewTimer(time.Duration(globalNewAssessmentCoolOff) * time.Millisecond).C
+			}
+
 			if moreAssessments {
-				if activeAssessments < maxAssessments {
+				if currentAssessments < maxAssessments {
 					host, hasNext := manager.hostProvider.next()
 					if hasNext {
 						manager.startAssessment(host)
@@ -812,9 +876,16 @@ func main() {
 	var conf_usecache = flag.Bool("usecache", false, "If true, accept cached results (if available), else force live scan.")
 	var conf_maxage = flag.Int("maxage", 0, "Maximum acceptable age of cached results, in hours. A zero value is ignored.")
 	var conf_verbosity = flag.String("verbosity", "info", "Configure log verbosity: error, notice, info, debug, or trace.")
+	var conf_version = flag.Bool("version", false, "Print version and API location information and exit")
 
 	flag.Parse()
 
+	if *conf_version {
+		fmt.Println(USER_AGENT)
+		fmt.Println("API location: " + apiLocation)
+		return
+	}
+
 	logLevel = parseLogLevel(strings.ToLower(*conf_verbosity))
 
 	globalIgnoreMismatch = *conf_ignore_mismatch
@@ -881,6 +952,10 @@ func main() {
 			var results []byte
 			var err error
 
+			if hp.StartingLen == 0 {
+				return
+			}
+
 			if *conf_grade {
 				// Just the grade(s). We use flatten and RAW
 				/*