Using JSON line format for QueryCapture #193
Conversation
| return nil, err | ||
| } | ||
| // read existing queries from file | ||
| handler.ReadAllQueriesFromFile() |
There was a problem hiding this comment.
handle error and close file if err != nil
| log.WithError(ErrUnexpectedCaptureChannelClose).WithField(logging.FieldKeyEventCode, logging.EventCodeErrorCensorSecurityError) | ||
| } | ||
|
|
||
| // TODO: how to remove duplicate code? |
There was a problem hiding this comment.
Golang doesn't provide way to write two cases at once like in switch with fallthrough becase any read from channel block goroutine and for fallthrough it need to read second channel :) So we can hack it like:
- move logic to some function
func (handler *QueryCaptureHandler) finish(){
handler.serializationTicker.Stop()
err := handler.DumpAllQueriesToFile()
if err != nil {
log.WithError(ErrComplexSerializationError).WithField(logging.FieldKeyEventCode, logging.EventCodeErrorCensorSecurityError)
}
openedFile.Close()
}
and call it in two cases
case <-someCh:
....
handler.finish()
return
....
- use one more channel that will flag that we need to finish
exitCh := make(chan bool)
for {
select {
case <-signalBackgroundExit:
exitCh <- true
case <-signalShutdown:
exitCh <- true
}
select {
case <-exitCh:
// exit code
default:
// do nothing, continue
}
}
| default: //channel is full | ||
| log.Errorf("Can't process too many queries. Skip query: %s", trimToN(query, LogQueryLength)) | ||
| } | ||
| handler.BufferedQueries = append(handler.BufferedQueries, *queryInfo) |
There was a problem hiding this comment.
better to append pointers instead copy of struct. change type of BufferedQueries from QueryInfo to *QueryInfo and then append only pointers.
| func (handler *QueryCaptureHandler) Serialize() error { | ||
| jsonFile, err := json.Marshal(handler.Queries) | ||
| func (handler *QueryCaptureHandler) DumpAllQueriesToFile() error { | ||
| // open or create file, NO APPEND |
There was a problem hiding this comment.
what about to truncate file and add O_TRUNC flag too?
|
|
||
| // clean buffered queries | ||
| handler.BufferedQueries = nil | ||
| err := AppendQueries(buf, openedFile) |
There was a problem hiding this comment.
we can just return AppendQueries(buf, openedFile)
or return err below without if
| return nil, err | ||
| } | ||
| if len(jsonQueryInfo) > 0 { | ||
| jsonQueryInfo = append(jsonQueryInfo, "\n"...) |
There was a problem hiding this comment.
can we change to append(jsonQueryInfo, '\n') (single quotes are one character literal, double quotes are string literal that iterable)? after `"\n"... I think about "is it only 1 character or it 2 characters and therefore we use ... ?" )
|
@Lagovas thank you for your comments take a look! |
As we discussed previously having large JSON file as Query blob might be not the best option [T605]