In linguistics, how do you know when to use the brackets [ ] vs. slashes / / ?

Slashes are used when the sound in question is a phoneme and in a "broad" transcription, while square brackets are used for allophones or for phonetic, "narrow" transcriptions (not phonemic transcriptions). I'll illustrate with an example: 

In English, /p/ and /b/ are both phonemes, as we can create minimal pairs where the substitution of one with the other creates a difference in meaning. Hence, the only difference between /pæt/ and /bæt/ is the /p/ and the /b/, which have made different words with different meanings. 

If, however, we want to be a little bit more specific when describing the sound quality of /p/, for example, we can say that in word-initial position, /p/ becomes 'aspirated' (ie. produced with a puff of breath). It then becomes the allophone [pʰ]. This allophone is not a phoneme, because the use of [pʰ] as opposed to /p/ does not create a difference in meaning - it simply describes the quality of the sound. 

Dictionaries include broad phonemic transcriptions, but when we're describing someone's speech, we want to be more specific; the use of [ ] shows that we are using a phonetic, not a phonemic, transcription. 

And by the way, /n/ becomes velarised, not palatalised, before /k/ and /g/ (a palatal /n/ is the 'ny' sound in 'Sonya'). The rules for this phenomenon would be something like: 

V --> [+velar]/[+velar] 
C --> [+velar]/[+velar] 

(The rule is generally expressed each time for either a vowel or a consonant, I believe - I'm not sure if one can use brace notation with the V/C). 

In other words, a vowel/consonant becomes velarised when it precedes a velar consonant, such as /k/ or /g/. In this case, /n/ becomes /ŋ/ before /k/ or /g/.